Photos of our building process.

Serial Communication Tutorials

These tutorials are eye-opening!

Here is a link to my ‘homework’ video:

For the first Call/Response exercise I used two different sensors – tilt and light, with ‘wonky’ results. For the second Call/Response I used two of the same – both light sensors – with much smoother, less erratic results.

Unfortunately my trial copy of Max6 has expired and I wasn’t able to try the patches . . ..

John

The Lightness of Sound

By Jeremy Littler and Mitzi Martinez

What is The Lightness of Sound?

SEE The Lightness of sound:

 

Based in the LUMARCA design created by Albert Hwang, Matt Parker and Eliot Woods (http://lumarca.info/), The Lightness of Sound is an installation designed to visualize music in a 3D light display, adding a new dimension to the interaction of the musicians with their performance, by adding a visual element displayed in a physical object that reacts on every note that is being played.

 For the musicians, this completes the sensorial experience as by playing they are physically touching the instrument to produce notes, they hear it, and they see the volumetric display in the WireMap reacting to their input by showing changes of length and color of the light.

 For the spectators, it also adds a new dimension as they can see the synergy between music, instrument and musician condensed in a physical object that displays this interactions and the flow of the music in a very visual and almost tactile way.

How it works?

On the technical side, the Lightness of Sound project goal, was to integrate a wireless MIDI system (in this case a MIDI guitar) with a projected visualization. Ideally, live music could be performed that seamlessly blended both the visual and auditory elements into a seamless experience for the audience.  In theory, the wireless MIDI system would allow musicians to create a dynamic musical compositions influenced the projected content, while at the same time being influenced musically by visualizations.  The core elements of the wireless MIDI system were as follows:

1)   A Midi Guitar with support for battery based operation.

2)   An Arduino with MIDI input capabilities.

3)   An XBee based transmission system (for MIDI).

4)   A Processing application that integrated audio and projection playback functions.

5)   A virtual MIDI system that enabled Processing to convert MIDI notes to sound.

MIDI OUTPUT (FOR INFORMATIONAL PURPOSES)In terms of the hardware/software requirements Arduinos are easily configured for MIDI output  (See example below).

 

 The hardware requirements are a MIDI Connector  (PRT-09536 PRT-09536 from Sparkfun), three segments of wire and a 220 ohm resistor.  To output MIDI, you can install the RWMidi Library for processing or MidiBusMIDI content generated by Processing can be patched via Apple’s IAC Driver to Garage Band (or other sound generators) or connected to a USB-Midi interface (e.g., the Uno) and then taken as an input in the sound generator. There is an example of this approach at Instructables. The process of configuring an Arduino for MIDI input, as was required by this project, is more complex…

MIDI INPUT

Developing a solution based on MIDI input is more complex than MIDI output. There are essentially two ways to configure an Arduino as a MIDI input device. The least costly option is to breadboard a MIDI IN interface (see below) using basic electronic components.

Source: Notes and Volts:

http://www.notesandvolts.com/2012/01/fun-with-arduino-midi-input-basics.html

 

The parts requirements for this project are:

 

6N138 Optocoupler

A 270 ohm resistor

A 220 ohm resistor

A 1N914 Diode

A Midi Socket

3 wires.

The total cost of these items is under $10.00.

Note: If connected, this circuit will conflict with the programming of the Arduino. Therefore, the circuit must be disconnected when performing this task. It would be worth installing a “programming” switch to simplify development.

Despite the circuit being fairly simple, Jeremy spent an entire day attempting to get it to work (see below) without success. Simply put, the circuit would not communicate MIDI information to the Arduino. Perhaps others will have more success with it. It’s worth noting that there were comments online that others were unable to get this to function. It’s possible that changes to the Arduino environment or hardware have rendered this circuitry inoperative.

The second method (as used in this project) of enabling MIDI IN on an Arduino is to attach the SparkFun MIDI Shield to the Arduino (PRT-09595 from SparkFun). However, it’s worth noting that this module (which handily includes a programming/running switch) does block access to the RX/TX headers on the Arduino. This presents a challenge when combining MIDI Input with an XBee via an Arduino.

To receive notes from the MIDI shield an Arduino library is required. Specifically, the Arduino Midi Library (version 3.2+).  As the MIDI shield overrides the serial communications sub-system on the Arduino it is not possible to simultaneously use a MIDI shield and an XBee for serial communication using standard serial techniques. To overcome this limitation, the Arduino can be configured to communicate with the XBee using the SoftWare Serial Library.  However, this does limit the baud rate of the XBee communication to a maximum of 9600bps.

Source: SparkFun


The Arduino code for the MIDI IN project is as follows:

//*******************

// Wireless Midi to Xbee

// By Jeremy Littler

// Using 2 Key Libraries. See below

// Note: on the Yamaha - EZ-AG the Note Range it is 41-76

// ****************

//

//                            ==================

//   ooo                   ===================

//   ===               ===============

//   ====================[]===[]=====   ROCK ON!

//   ===              ================

//   ooo                 ==================

//                             ===============

// THE ESSENTIAL

// SoftSerial Library from Arduino

// http://arduino.cc/en/Reference/SoftwareSerial

// THE ESSENTIAL

// Arduino Midi Library from Ardiono

//http://www.arduino.cc/playground/Main/MIDILibrary

// fortyseveneffects (at) gmail (dot) com.

// To overcome the Sparkfun midi adapter taking

// Over the Flipping Physical Serial Port. use SoftwareSerial

// to communicate with the XBee

//Don't Forget to That the Midi Shield Needs to

// Be Manually Switched (near the midi plugs)

// Between Program and Operation Mode

#include <SoftwareSerial.h>

#include <MIDI.h>  // Add Midi Library

// LED is defined to determine (easily) the MIDI Value

#define LED 13    // Arduino Board LED is on Pin 13

// Use Software Serial to talk to the Xbee. Whoppeeeee!

// This used Pin 11 to Output the Midi Value

// Connect the Jumper Wire from

// From Pin 11 on the Arduino to the

// XBee Station

SoftwareSerial myXbeeSerial(10, 11); // RX, TX

String MidiNoteString = "02#";

//  Channel (Ignored), Pitch (Kept), and Velocity (Ignored)

void MyHandleNoteOn(byte channel, byte pitch, byte velocity) {

// Have Received Data so Prepare it to be printed to the XBEE

String MidiNoteString = "02#" + String(pitch);

if (velocity == 0) {//A NOTE ON message with a velocity = Zero is actualy a NOTE OFF

digitalWrite(LED,LOW); //Turn LED off

// THE 01 VALUE DEMARKS MIDI DEVICE 1

// THE 02 VALUE DEMARKS MIDI DEVICE 2

String MidiNoteString = "02#" + String("999");

myXbeeSerial.println(MidiNoteString);

} else {

digitalWrite(LED,HIGH);//Turn LED off

// THE 01 VALUE DEMARKS MIDI DEVICE 1

// THE 02 VALUE DEMARKS MIDI DEVICE 2

myXbeeSerial.println(MidiNoteString);

}

}

void setup() {

myXbeeSerial.begin(9600);

pinMode (LED, OUTPUT); // Set Arduino board pin 13 to output

// OMNI sets it to listen to all channels.. MIDI.begin(2) would set i

MIDI.begin(MIDI_CHANNEL_OMNI);  // Initialize the Midi Library.

// Kill Thru Mode as it hogs performance

MIDI.turnThruOff();

// Don't Need Sysex Message so turn that off

MIDI.disconnectCallbackFromType(SystemExclusive);

//Register a Function to Call Note on/off Messages

MIDI.setHandleNoteOn(MyHandleNoteOn);

}

void loop() { // Main loop

MIDI.read(); // Continually check what Midi Commands have been received.

//myXbeeSerial.println("Hello, world?");

}

XBEE CONFIGURATION

 Two XBee units are configured using CoolTerm (in OS/X) with the following settings:

XBee Receiver:

Pan ID: 2171 - ATID2171 (XBee 1 and 2 must have the same Pan ID).

ATDL 0

ATMY: 1

XB (MIDI) Sender:

Pan ID: 2171 - ATID2171

ATDL: 1

ATMY: 0

Both Xbee units should be set to a data rate of 9600 BPS (ATBD3 in CoolTerm)

 One XBee is wired to the headers on the SparkFun MIDI shield, as follows:

  • Pin 11 to Pin 3 on the XBee
  • +5V to Pin 1 on the XBee
  • GND to GND on the XBee

We worked over the Processing Sketch created by Matt Parker, changing the display and combining the MIDI Read Functionality and the Project visualization that we used is the following:

 

/*

Wiremap Renderer for 2 Globes
by

/ \
______/________\______
/ \ / \
/________\ /________\______
/ \ | \
/ / \ \_____ | \
/________\______ | /
\ / \ |_____/
\ /________\
\ \ /
\_____ \ /
/ \
/ \

For more information on the project please visit:
http://wiremap.phedhex.com

This program builds two separate 3d globes. I have two separate functions (& sets of variables) because I haven't quite yet figured out how call a function twice. Elementary stuff, I know, but I'll get to it when I can.

Some conventions to be aware of:

1 - This particular program builds in millimeters.
2 - Also, I use a left-handed coordinate system, with positive X going right, positive Y going up, and positive Z going forward.
*/

// Fullscreen stuff:

/* Variable declarations
---------------------------------------------------------*/

/* Physical Wiremap, in inches
---------------------------------------------------------*/

float depth = 70.0; // The mapline lies 3 meters away from the projector's focal point
float map_length = 32.0; // The mapline is 1.28 meters wide
float depth_unit = 0.500; // Each depth unit is 5 millimeters
float map_unit = 0.7; // Each mapline unit is 5 millimeters
int wire = 64; // There are 128 wires in this Wiremap
float depth_thickness = 30.0; // How deep is the field (perpendicular to the mapline)
/* Projector
---------------------------------------------------------*/

float ppi = 40; // Pixels per millimeter (unit conversion). Only true for mapline plane - 4 pixels every 5 millimeters
int string_pix_count = 9; // How many columns of pixels are being projected on each string //Mitzi's comments: here I changed the thickness of the lines.
/* Map
---------------------------------------------------------*/

float[] map = new float[wire]; // example: map[0] = 90 means that the first string is 45 cm away from the mapline
float[] x_by_ind = new float[wire]; // x coordinate for each wire
float[] z_by_ind = new float[wire]; // z coordinate for each wire
/* Globe A
---------------------------------------------------------*/

float[] globe = new float[3]; // globe x,y,z coords
float radius = 31.00; // default globe radius
int dot_height = 15; // height of surface pixels.
boolean render_globe = true; // toggle globe rendering

/* Key input
---------------------------------------------------------*/

float step = .2; // how far the globe moves / button press
boolean mouse = true; // is mouse clicked?
int colorval_r = 200; // red
int colorval_g = 100; // green
int colorval_b = 255; // blue
boolean xpin = false; // the mouse controls the globe's y & z axis
boolean ypin = true; // x & z
boolean zpin = false; // x & y
int start_time = 0; // for beat mapper
int end_time = 0; //
float beat_multiplier = 1; // multiplies freqeuncy of how often beat hits

 

/* Beat Mapper Variables
---------------------------------------------------------*/

int[] last_32 = new int[32]; // last 32 times the spacebar has been pressed
int times_struck; // nubmer of times spacebar struck since timer was reset
int first_strike; // millis value for when timer was reset
int period = 500; // time between beats (this is the metronome)
int offset = 1; // how far away in time we are from the last beat
/* wave variables
---------------------------------------------------------*/
int trail = 350; // number of iterations of past mouse clicks we keep
int[] click_time = new int[trail]; // array of times (millis) associated w/ clicks
int[] click_x = new int[trail]; // array of x locales for clicks
int[] click_y = new int[trail]; // array of y locales for clicks
float[] click_x_trans = new float[trail]; // translations from mouse x to xyz
float[] click_y_trans = new float[trail]; // translations from mouse y to xyz
float amplitude = .6; // amplitude of waves
int decay = 3000; // how long it takes for waves to die down
float wave_velocity = .035; // inches/milliseconds
int trail_frequency = 10; // milliseconds - NOT frequency of wave, but how often a new value gets pushed into the trail arrays (above)
int trail_cycle_count; // this gets updated once every (trail_frequency)
int trail_cycle_count_compare; // this is used to check to see if we need a new value
int water_color = 0; //

float plane_angle = .0; // the angle of the plane of the water (think m in y = mx + b)
float plane_intercept = 0; // where the plane intersects the origin (think b in y = mx + b)
import processing.opengl.*;
// *******
// JL CODE
// ******

// ********************
// CODE TO READ SERIAL PORT FROM XBEE
// AND CONVERT TO VIRTUAL MIDI
// **************** *
import rwmidi.*;
MidiOutput output;
import processing.serial.*;
Serial myPort; // Create object from Serial class
String val ="0"; // Data received from the serial port
String CurrentNote = "0"; // Current Note in the Serial Buffer
String LastNotePlayer = "0"; // Last Note Played
int PitchRanged = 0; // Converts the Midi Note Value to a More Predicatable Range Value

String inBuffer = ""; // Serial Port Value Received
String postinBuffer; // Used to Time WhiteSpace entc
String line;
// Used to Control the Color of the Tips in the Project Window
int tipR = 200;
int tipG = 70;
int tipB = 205;
float tipvalue=0;
int GlobeTip = 255;

// Controls the Globe Movement Accross the X Axis
float midi_mousex = 0;
// Controls the Globe Movement Accross the Y Axis
float midi_mousey = 0;
// Total Notes Played. This creates an iterative environment
int totalnotesplayed = 0;
// Controls
float midi_mouse_glide_x = 0;
float midi_mouse_glide_y = 0;

// END JL CODE *******

 

 

static public void main(String args[]) {
PApplet.main(new String[] { "--present", "olasverdes" });
}

void setup() {

// JL START CODE
// List The Serial Devices
println(Serial.list());

// Soft Serial can only send at 9600 safely. Therefore, the Xbee has
// Been Configured to 9600 Baud.
myPort = new Serial(this, Serial.list()[0], 9600);
myPort.bufferUntil('\n');
// Configures The System for Midi
// creates a connection to IAC as an output
output = RWMidi.getOutputDevices()[0].createOutput(); // talks to garageband
// END JL CODE

// JL - HAD TO REMOVE OPEN GL COMMENT
size(displayWidth, displayHeight);
//size(displayWidth, displayHeight, OPENGL);
background(255);
colorval_r = 25; //Mitzi's notes: color of the globe that floates over the wave
colorval_g = 200;//Mitzi's notes: color of the globe that floates over the wave
colorval_b = 100;//Mitzi's notes: color of the globe that floates over the wave
loader();
}

void draw() {
noCursor();
noStroke();
frameRate(30);
fill(0); //Mitzi's notes: this is the background color
rect(0, 0, width, height);

sineSurface();
}

void sineSurface() {

/* trail_frequency appends clicks to the mouse trail arrays */

int remainder = millis() % trail_frequency;
trail_cycle_count = (millis() - remainder) / trail_frequency;
if (trail_cycle_count != trail_cycle_count_compare) {
trail_cycle_count_compare = trail_cycle_count;

// JL Changed
append_click(int(midi_mouse_glide_x), int(midi_mouse_glide_y));

// append_click(mouseX, mouseY);
}

float[] time_since_click = new float[trail]; // the difference between now and the array of clicks
float[] amp_modifier = new float[trail]; // the amp according to decay and time since click
float[] distance_since_pass = new float[trail]; // the distance since the head of the wave has passed the string
float[] distance_since_pass_fraction = new float[trail]; // the distance gets multiplied by a fraction for beat mapping (period)
float[] time_since_pass = new float[trail]; // amount of time that has passed since head of wave & wire intersection
float[] wave_head_distance = new float[trail]; // distance between epicenter and head of wave
float[] amp = new float[trail]; // amplitude of wave @ wire point according to mouse movement & beatmapping

/* for each wire... */

for(int i=0; i<wire; i+=1) {
float final_amp = z_by_ind[i]*plane_angle + plane_intercept ; // the baseline for the final amplitude is an upward slope when looking @ the wiremap... used y = mx + b

for(int x = 0; x < trail; x ++ ) {

float local_hyp = sqrt(sq(x_by_ind[i] - click_x_trans[x])+sq(z_by_ind[i] - click_y_trans[x]));
time_since_click[x] = millis() - click_time[x];
wave_head_distance[x] = time_since_click[x] * wave_velocity;
distance_since_pass[x] = wave_head_distance[x] - local_hyp;
distance_since_pass_fraction[x] = distance_since_pass[x] / float(period / 6);
time_since_pass[x] = distance_since_pass[x] / wave_velocity;
if (time_since_pass[x] > 0 && time_since_pass[x] < decay ) {
amp_modifier[x] = time_since_pass[x] / decay - 1;
}
else {
amp_modifier[x] = 0;
}
amp[x] = - amplitude * amp_modifier[x] * sin((2 * PI * distance_since_pass_fraction[x]));

final_amp = final_amp + amp[x];
}

float y_top_coord = final_amp;
float y_bot_coord = -20;
float y_top_proj = y_top_coord * depth / z_by_ind[i]; // compensate for projection morphing IN INCHES
float y_bot_proj = y_bot_coord * depth / z_by_ind[i];
float y_height_proj = y_top_proj - y_bot_proj;
// JL Modified - Added Variables for TipR, TipG and TipB
fill(tipR,tipG,tipB); //Mitzi's notes: here you change the color of the tip in every line // draw a rectangle at that intersect

//int tipR = 200
//int tipG = 70
//int tipB = 205
// rect 1 is top dot for sliver
float left1 = i * (width) / wire;
float top1 = (height/ppi - y_top_proj) * ppi;
float wide1 = string_pix_count;
float tall1 = dot_height;
rect(left1, top1, string_pix_count, tall1); // draw a rectangle at that intersect

// rect 3 is filler for sliver
fill(200, 255, water_color); // Mitzi's notes: here you change the color of the wave.

float left3 = i * (width) / wire;
float top3 = (height/ppi - y_top_proj) * ppi + dot_height;
float wide3 = string_pix_count;
float tall3 = y_height_proj * ppi - (dot_height * 2);
rect(left3, top3, string_pix_count, tall3); // draw a rectangle at that intersect
}

// Jeremy Changed Code
float globe_x = (midi_mousex / float(width)) * (map_length) - (map_length / 2);
// float globe_x = (mouseX / float(width)) * (map_length) - (map_length / 2);
float globe_z = depth - (midi_mousey) / float(height) * (depth_thickness);
// float globe_z = depth - (mouseY) / float(height) * (depth_thickness);

float y_amp = globe_z*plane_angle + plane_intercept;
for(int x = 0; x < trail; x ++ ) {

float local_hyp = sqrt(sq(globe_x - click_x_trans[x])+sq(globe_z - click_y_trans[x]));
time_since_click[x] = millis() - click_time[x];
wave_head_distance[x] = time_since_click[x] * wave_velocity;

distance_since_pass[x] = wave_head_distance[x] - local_hyp;
distance_since_pass_fraction[x] = distance_since_pass[x] / float(period / 6);
time_since_pass[x] = distance_since_pass[x] / wave_velocity;
if (time_since_pass[x] > 0 && time_since_pass[x] < decay ) {
amp_modifier[x] = - time_since_pass[x] / decay + 1;
}
else {
amp_modifier[x] = 0;
}
amp[x] = - amplitude * amp_modifier[x] * sin((2 * PI * distance_since_pass_fraction[x]));

y_amp = y_amp + amp[x];
}
float globe_y = y_amp;
float radius = (sin(TWO_PI * float((millis() - offset) % period) / float(period)) + 1) / 2;
if(render_globe == true) {
gen_globe(globe_x, -globe_y, globe_z, 5);
}
//println(globe_x + " " + globe_y);
if (millis() > 10000) {
}
}
void append_click(int local_mouseX, int local_mouseY) {
click_time = subset(click_time, 1);
click_x = subset(click_x, 1);
click_y = subset(click_y, 1);
click_x_trans = subset(click_x_trans, 1);
click_y_trans = subset(click_y_trans, 1);
click_time = append(click_time, millis());
click_x = append(click_x, local_mouseX);
click_y = append(click_y, local_mouseY);
click_x_trans = append(click_x_trans, (local_mouseX / float(width)) * (map_length) - (map_length / 2));
click_y_trans = append(click_y_trans, depth - (local_mouseY) / float(height) * (depth_thickness));
}
void gen_globe(float x, float y, float z, float rad) {
for(int i = 0; i < wire; i += 1) {
if((x_by_ind[i] >= (x - rad)) && (x_by_ind[i] <= (x + rad))) { // if a wire's x coord is close enough to the globe's center
float local_hyp = sqrt(sq(x_by_ind[i] - x) + sq(z_by_ind[i] - z)); // find the distance from the wire to the globe's center
if(local_hyp <= rad) { // if the wire's xz coord is close enough to the globe's center
float y_abs = sqrt(sq(rad) - sq(local_hyp)); // find the height of the globe at that point
float y_top_coord = y + y_abs; // find the top & bottom coords
float y_bot_coord = y - y_abs; //
float y_top_proj = y_top_coord * depth / z_by_ind[i]; // compensate for projection morphing
float y_bot_proj = y_bot_coord * depth / z_by_ind[i];
float y_height_proj = y_top_proj - y_bot_proj;
/* Top dot
---------------------------------------------------------*/
fill(colorval_r, colorval_g, colorval_b); // Mitzi's notes: here you can change the color of the globe as well. Fill the globe pixels this color
float left1 = i * (width) / wire;
float top1 = (height/ppi - y_top_proj) * ppi + dot_height; // ppi = pixel / mm. These are conversions to & from pixels and mm
float wide1 = string_pix_count;
float tall1 = y_height_proj * ppi - (dot_height * 2);
rect(left1, top1, wide1, tall1);
fill(GlobeTip); // Mitzi's notes: here you change the color of the tip of the globe. If you put fill(255) it will be white.

/* Top Surface
---------------------------------------------------------*/
float left2 = i * (width) / wire;
float top2 = (height/ppi - y_top_proj) * ppi;
float wide2 = string_pix_count;
float tall2 = dot_height;
rect(left2, top2, wide2, tall2);

/* Bottom Surface
---------------------------------------------------------*/
float left3 = i * (width) / wire;
float top3 = (height/ppi - y_bot_proj) * ppi - dot_height;
float wide3 = string_pix_count;
float tall3 = dot_height;
rect(left3, top3, wide3, tall3);
}
}
}
}

 

void mousePressed() {
if (mouseButton == LEFT) {
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
} else if (mouseButton == RIGHT) {
if (water_color == 255) {
water_color = 0;
} else {
water_color = 255;
}
}
}

void keyPressed() {

/* Globe A
---------------------------------------------------------*/
if (true == true) {
if (key == 'w') { // adds value to the dimension that the mouse cannot move in
if (xpin == true) {
globe[0] = globe[0] + step;
} else if (ypin == true) {
globe[1] = globe[1] + step;
} else if (zpin == true) {
globe[2] = globe[2] + step;
}
} else if (key == 's') {
if (xpin == true) { // subtracts value from the dimension that the mouse cannot move in
globe[0] = globe[0] - step;
} else if (ypin == true) {
globe[1] = globe[1] - step;
} else if (zpin == true) {
globe[2] = globe[2] - step;
}
} else if (key == 'e') { // adds to radius
radius = radius + step;
} else if (key == 'd') { // subs from to radius
radius = radius - step;
} else if (key == 'a') { // allows mouse control for radius (hold down 'a' and bring mouse up or down)
radius = (height - mouseY) * .8;
mouse = false;
} else if (key == 'q') { // stops ball in place so that you can pop it somewhere else
mouse = false;
} else if (key == 'z') { // color control (hold down buttons and bring mouse up or down)
colorval_r = (height - mouseY) * 255 / height;
} else if (key == 'x') {
colorval_g = (height - mouseY) * 255 / height;
} else if (key == 'c') {
colorval_b = (height - mouseY) * 255 / height;
} else if (key == 'v') {
colorval_r = (height - mouseY) * 255 / height;
colorval_g = (height - mouseY) * 255 / height;
colorval_b = (height - mouseY) * 255 / height;
} else if (key == '1') { // x y z pin switches
xpin = true;
ypin = false;
zpin = false;
} else if (key == '2') {
xpin = false;
ypin = true;
zpin = false;
} else if (key == '3') {
xpin = false;
ypin = false;
zpin = true;
} else if (key == 't') { // beat mapper buttons - start, stop, effects, and multipliers
start_time = millis();
} else if (key == 'y') {
end_time = millis();
period = end_time - start_time;
offset = start_time % period;
} else if (key == 'g') {
beat_multiplier = 1;
} else if (key == 'h') {
beat_multiplier = 2;
} else if (key == 'j') {
beat_multiplier = 4;
} else if (key == 'k') {
beat_multiplier = 8;
} else if (key == 'b') {
if (render_globe == false) {
render_globe = true;
} else {
render_globe = false;
}
}
}
}
void keyReleased()
{
if(mouse == false) {
mouse = true;
}
if(key==' ') {
if(millis()-last_32[31] > 1500) {
last_32[31] = 0;
}
last_32 = subset(last_32, 1);
last_32 = append(last_32, millis());
for(int i=31; i>=0; i--) {
if(last_32[i] == 0) {
times_struck = 31 - i;
first_strike = last_32[i+1];
break;
} else {
times_struck = 32;
first_strike = last_32[0];
}
}
if(times_struck > 1) {
period = (last_32[31] - first_strike) / (times_struck - 1);
}
offset = last_32[31];
}
}
void loader() { // loads data for this particular wiremap
map[0] = 15;
map[1] = 13;
map[2] = 0;
map[3] = 29;
map[4] = 37;
map[5] = 6;
map[6] = 31;
map[7] = 14;
map[8] = 9;
map[9] = 0;
map[10] = 12;
map[11] = 24;
map[12] = 3;
map[13] = 26;
map[14] = 39;
map[15] = 18;
map[16] = 3;
map[17] = 28;
map[18] = 11;
map[19] = 18;
map[20] = 1;
map[21] = 20;
map[22] = 24;
map[23] = 8;
map[24] = 7;
map[25] = 22;
map[26] = 17;
map[27] = 34;
map[28] = 37;
map[29] = 1;
map[30] = 23;
map[31] = 10;
map[32] = 2;
map[33] = 33;
map[34] = 6;
map[35] = 34;
map[36] = 27;
map[37] = 12;
map[38] = 19;
map[39] = 25;
map[40] = 11;
map[41] = 14;
map[42] = 5;
map[43] = 15;
map[44] = 27;
map[45] = 4;
map[46] = 25;
map[47] = 8;
map[48] = 32;
map[49] = 35;
map[50] = 7;
map[51] = 30;
map[52] = 21;
map[53] = 4;
map[54] = 16;
map[55] = 2;
map[56] = 20;
map[57] = 17;
map[58] = 38;
map[59] = 22;
map[60] = 32;
map[61] = 36;
map[62] = 30;
map[63] = 10;

for(int i = 0; i < trail; i ++ ) {
click_time[i] = 0 - (i * 500);
}
for(int j=0; j<wire; j++) { // calculate x and z coordinates of each wire
float xmap = (0 - (map_length / 2)) + j*map_unit;
float hyp = sqrt(sq(xmap) + sq(depth));
z_by_ind[j] = depth - map[j]*depth_unit;
x_by_ind[j] = xmap - xmap*map[j]/hyp*depth_unit;
}
}

 

//***********
// JL CODE TO GRAB MIDI INPUT
// FROM XBEE
//**************
void serialEvent (Serial myPort) {

try {

// The Midi Guitar - EZ AG I am using
// Produces Midi Note Values from a
// Total Range of
// Open E = 52
// High E on 6th String = 88
// You will need to adjust the values (i.e. using the range command possible).
// To reflect your particular midi device
// 1st (Low) 52 Open - 64
// 2nd 57 Open to 69
// 3rd 62 open to 74
// 4th 67 open to 79
// 5th 71 open to 83
// 6th 76 open to 88
inBuffer = myPort.readStringUntil('\n');
postinBuffer = trim(inBuffer);
CurrentNote = postinBuffer.substring(3,5);
//CurrentNote = trim(CurrentNote);
// send a note 200 mS long
int duration = 200;
int note = 0;
int channel = 1; // GB seems to ignore this, but mehtod needs it

// Definately Wrap this Code or there may be catastrophic exceptions of the serial port gets overloaded

try {

if (CurrentNote == null) {
return;
} else {

if (CurrentNote.equals("99")) {
int velocity = 0;
note = 0;
int success = output.sendNoteOn(channel, note, velocity); // sendNoteOn returns 0 = fail, 1 = success
delay(duration); // note length -see keys below for better solution
success = output.sendNoteOff(channel, note, velocity); // sendNoteOff returns 0 = fail, 1 = success
note = 0;

} else {

// This is protected by a try box to ensure a never lock situation

try {

note = int(CurrentNote); // midi notes are numbered 0 -127 -- not all notes are played by all voice
LastNotePlayer = (CurrentNote); // Store the last note in a variable
int velocity = 127; // range 0-127, 127= max
int success = output.sendNoteOn(channel, note, velocity); // sendNoteOn returns 0 = fail, 1 = success
delay(duration); // note length -see keys below for better solution
success = output.sendNoteOff(channel, note, velocity); // sendNoteOff returns 0 = fail, 1 = success
totalnotesplayed = totalnotesplayed + 1;
// Get the Integer Value of the Note
//println(note);

} catch (Exception e) {
}

try {

// Sets the Color Value of the Tip
tipR = int(map(note, 52, 88, 150, 255));
tipG = int(map(note, 52, 88, 30, 90));
tipB = int(map(note, 52, 88, 0, 10));

// Displays Red at the Highest points of the scale
if (note >= 80) {
tipR = 255;
tipG = 0;
tipB = 0;
// Note Yellow Value = tipR = 255, tipG = 255, tipB = 0;
}
// Moves the Globe along the X-Axiz
midi_mousex = int(map(note, 52, 88, 500, 2100));
midi_mousex = midi_mousex + log(random(50));
midi_mousey = int(map(note, 52, 88, 200,600));
midi_mousey = midi_mousey + log(random(50));

// Sets the Ampltude of the Waves
// High Frequency notes have greater impact on amplitude
//if (note < 65) {
int ampMap = int(map(note, 52, 88, 0, 50));
amplitude = ampMap/50 + .9;

//} else {
//int ampMap = int(map(note, 52, 88, 0, 50));
//amplitude = ampMap/50 + .9 + random(0, .65);

//}
// Controls the X and Y movements of the Main Bars
midi_mouse_glide_x = int(map(note, 52, 88, 300, 500));
midi_mouse_glide_y = int(map(note, 52, 88, 100, 300));
midi_mouse_glide_x = midi_mouse_glide_x + random(200, 300);
midi_mouse_glide_y = midi_mouse_glide_y + random(200, 300);
// Color of the Globe
// This creates a Red Value. Impacted by Note Value
colorval_r = 255;
colorval_g = 0;
colorval_b = int(map(note, 52, 70, 255, 0));
// The Color of the Bars. Mapped from Note Value
water_color = int(map(note, 52, 88, 0, 255));

// Control the Decay Rate of the Bars. These are Mapped to the Note Value
// The total notes played also affects the decay value
int decay = int(map(note, 52, 88, 3000, 6000));
decay = decay - totalnotesplayed;

// Control the Wave Velocity
float wave_velocity = int(map(note, 52, 88, 0.035, 2));
int trail_frequency = int(map(note, 52, 88, 30, 50));

} catch (Exception e) {
}

}
}

} catch (Exception e) {
}

} catch (Exception e) {
}
}

//***********
// END JL CODE
//***********

 

CONNECTING IT ALL UP:

 The wireless MIDI system should be placed in a robust but portable project case (see example below):

The instrument selected for this project was a Yamaha EZ-AG MIDI Guitar. The EZ-AG is no longer in production. However, any MIDI instrument is a potential candidate, assuming that the device looks can be battery powered.

Processing is not a particularly flexible tool for sound playback scenarios. Therefore, the MIDI information will likely need to be “piped” using a virtual MIDI configuration (e.g., via Audio MIDI Setup in OS/X) to GarageBand or a similar MIDI capable DAW. The process for wiring GargeBand specifically (though this could apply to other audio applications) is presented in detail at Hex705.  This element was essential to getting the “Pad” playback sounds required for the Lightness of Sound project.

 

BUILDING THE WIREMAP

To build the WireMap for the display, Mitzi followed this instructable:

http://www.instructables.com/id/How-to-Build-a-Wiremap/

We wanted to create smaller versions of it but the wiring for them proved to be a real challenge, the smaller the version, the more difficult is to put a wire without moving the one next to it.

 Mitzi Built 2 prototypes before settling with the a final version that is slightly smaller than the one described in the instructable.

TECHNICAL CHALLENGES:

 The biggest limitation of the approach identified is that of latency, as the SoftwareSerial library limits the data rate to 9600 bps. Players must be careful to avoid overloading the MIDI buffer.  The next iteration of this project would determine if a custom-built MIDI input module would improve MIDI transmission speeds and, therefore, reduce latency issues. Building a custom module would significantly reduce the cost of each transmission system and, therefore, enable more instruments to be employed.

 During testing the wireless MIDI circuit performed flawlessly for over 4 hours. However, on the day of the presentation the device became extremely unreliable (Murphy’s Law!). This may have been due to the number of XBee units in the performance space, as there were a large number of projects being exhibited at the time.  The lesson from this experience is that wireless communications can be very unpredictable. To compensate for this attention should be paid to isolating the XBee units from cross frequency traffic by modifying the XBee channel settings (there are 16 frequency bands) and by verifying that the devices have unique Pan ID’s. Finally, as each XBee has a per-programmed serial number it is possible to add this information to the Processing code to verify that the data packets are coming from the correct XBee units. Finally, it would have prudent to add Cyclical Redundancy Checks (CRC)/Error Checking to the communication routines. This would eliminate spurious note data.

In terms of the WireMap, building in in core foam allowed us to move the wires to calibrate it more easily that it would have been if it would have been constructed in acrylic, however, the fragility of it also made it easy to bow, causing some of the wires to miss the light of the proyector.

 PAST AND FUTURE OF THE LIGHTNESS OF SOUND:

We started the project with the idea of building 2 of more WireMaps that could communicate to each other via XBee’s  and  react to user interaction, however, the difficulties on building smaller versions of it in core foam, brought us to a better concept, the idea of building a bigger one that could react to music.

For the future we foresee that the ability to implement a wireless MIDI approach opens up virtually unlimited performance possibilities. In addition to utilizing existing MIDI technologies it is possible to design entirely new processes for eliciting audience interaction.  The first step in exploring the performance options would be to investigate the range of MIDI instruments currently available. All MIDI instruments are considered to be “controllers”. However, the devices tend to be used primarily as instruments or as dedicated input controllers (see examples below):

While the above MIDI controllers are physical devices, it is possible to represent their functionality in virtual form (i.e., in code).  This is significant as physical MIDI devices are often expensive and have strict power requirements (e.g., many MIDI devices are not portable).

 

The option of implementing MIDI controllers on mobile devices (tablets in particular) significantly lowers the cost of providing large numbers of devices to “performers”. In addition, “virtual” MIDI devices can be programmed with innovative control surface designs. There are a many virtual MIDI SDK’s available that enable custom control surface modeling. At present, Jazz Mutant Lemur is considered to be amongst the most flexible (see below), but designers are free to choose form a very large palette of audio/midi design components. Virtual MIDI devices typically communicate performance information via the Open Sound Protocol (OSC) or via dedicated MIDI interfaces. However, MIDI’s compactness enables it to be easily routed through any communication protocol.

Finally, it’s possible to develop entirely novel instruments and control surfaces for musical interaction. Custom instruments can be simple or complex, standalone or interconnected. Instruments can even be implemented as sculptural elements.

Source: http://log.liminastudio.com/itp/neurohedron

PERFORMANCE GOALS:

 

Challenge:

 Is it possible to create a collective musical/visual experience by providing the audience with MIDI instruments?

 

The participatory audio concept for the Lightness of Sound Project is a physical environment where the line between audience and performers is blurred.  It is envisioned as a space where primary performers (i.e., a “band” or DJ’s) encourage their “audience” to create a collective music experience.  This real-time musical generative environment would be visibly projected. See below:

It is hoped that this environment would encourage the audience to create complex and evolving musical performances. The projected content would act as a key element in the feedback loop as it would encourage the audience to push themselves musically.

 

One of the concepts for the project was to implement a class of MIDI instrument based on didgeridoos. As this physical instrument requires significant practice (breathing control skills etc.) to become proficient, the concept was adapted to enable a more approachable and portable solution (see below):

Source: Jeremy Littler. Concept for Electronic Didgeridoo

Additional instruments would be developed to support rhythmic control (portable drum pads and triggers) and melodic performance instruments. These items would be distributed to the “audience” in the performance space.  The performers would also be encouraged to download an application to their portable devices that would enable them to contribute to both the sound scape and projected elements via a virtual MIDI controller. Finally, sensors would be installed to pick-up the audience’s motion, temperature etc. This information would alter the soundscape and the visualization at the same time.

Weatherbe (working title)

WEATHERBE – by Demi Kandylis, Ryan Rizzo and Heather Phenix

INSPIRATION

Our inspiration for the Weatherbe project was informed by nature, and the desire to render the weather into an artistic and comparative visualization of data from a specific set of cities across the globe. But why? For one, we found the important connection between weather and mood in humans to be intriguing, and the notion of how the relationship with one’s self can be deeply affected by weather patterns that are very much beyond our control. We thought about how weather is an increasingly common topic of conversation – often the basis of small talk – and more often the basis of important discussions surrounding disaster and the never-ending influx of opinions on global warming. It’s often the marker of how favourable a city is to live in, how we relate to each others’ state of being, whether together or apart, and how we decide on anything from which modes of transport to use, to how many layers to wear on a daily basis. Weather, in some ways, can be considered as central to how we live our lives as time.

References:
http://www.freewebs.com/partyhardinn/Relationship%20of%20Mood%20&%20Weather.pdf

When thinking about what kind of data visualizations could do justice to such a force, we realized that sometimes the most simple, uniform things can be the most effective. Our data sets needed to be representative of motion in an interesting way. We delved into creating mesmerizing simplicity of changing and interconnected shapes, and were inspired by Matthew and Susan Gorbet’s Chronos and Kairos project,  “a robotic suspended sculpture located at Gate 19 of the San Jose International Airport. An abstract timepiece. The choreographed movement of 65 suspended aluminum elements depict varying representations of time.”

Overall, we set out to play with temperature, wind speed, humidity and wind direction. We needed to create sketches that could build – increase and decrease – over time when fed weather API’s from cities of our choice…which brought us to our next question…which cities?

We thought in terms of our audience – the most relatively interesting cities would naturally be those of our classmates…displaying the mild, November humidity of Tehran, juxtaposed with the icy, snowy winds of Saskatoon, the dry desert heat of Beersheba, and the moderately cold Toronto would surely hit the mark.

Additional inspirations below:

Data vis- modern approaches
http://www.smashingmagazine.com/2007/08/02/data-visualization-modern-approaches/

Data vis processing code from Ben Fry (author of visualizing data)
http://benfry.com/writing/archives/3

Khan Academy
http://www.khanacademy.org/science/cosmology-and-astronomy

IBM think Exhibit Data Wall
http://www.behance.net/gallery/IBM-Think-Exhibit-Data-Wall/4619911?utm_source=Triggermail&utm_medium=email&utm_term=ALL&utm_campaign=MIH%20-%20November%2012

Amon Tobin’s show
http://www.youtube.com/watch?v=WWai4UZ0OqI

OPEN PROCESSING FAVES (H)

Curtain (like Harjot’s Kinect project)
http://www.openprocessing.org/sketch/20140

Tree
http://www.openprocessing.org/sketch/10318

Blobs
http://www.openprocessing.org/sketch/17043

Dough
http://www.openprocessing.org/sketch/65193

PROCESS

In creating the Weatherbee project, with the assigned wireless criteria in mind, we set out first to work together on creating something simple, that could be represented wirelessly through various forms of digital and physical light. The digital came first. Ryan and I discussed with Demi that our main goal for the project was to contribute to the coding, since this was not our strength and it was his. We discussed the use of projected light on mirrors, and looked at various examples of code that we found interesting to play with:
http://www.openprocessing.org/sketch/78479

http://www.openprocessing.org/sketch/25108

http://www.openprocessing.org/sketch/16965

http://www.openprocessing.org/sketch/5386

http://www.openprocessing.org/sketch/65193

Things got particularly frustrating when it came to the code. Ryan and I could create simple shapes that moved, but for whatever the reasons were, we never got to the point of implementing our work into the final product. Little did we know, Demi had been toiling away on a visualization that seemed to check the boxes for all of us in terms of its aesthetics. Here is the code he used:

CODE 

class Particle {
PVector location;
PVector velocity;
PVector acceleration;
float lifespan;
float wind;
PVector dir;
float temp;
float hum;
color col;

Particle(PVector l, float curLifespan) {
acceleration = new PVector(0,0.07);
velocity = new PVector(random(-1,1),random(-

2,0));
location = l.get();
lifespan = curLifespan;
}void run(WeatherObject wo) {wind = map(wo.wind,0,100,0,.5);
dir = wo.dir;
temp = wo.temp;
hum = wo.hum;
col = wo.col;update();
display();
}void update() {location.add(dir);
velocity.add(new PVector(wind*dir.x,wind*dir.y));
location.add(velocity);
lifespan -= 2.0;
}void display() {
fill(col,lifespan);
ellipse(location.x,location.y,2,2);
}boolean isDead() {
if (lifespan < 0.0) {
return true;
} else {
return false;
}
}
}class ParticleSystem {
ArrayList<Particle> particles;
float curLifespan = 400;

ParticleSystem() {
particles = new ArrayList<Particle>();
}

void addParticle() {
particles.add(new Particle(new PVector(width/2,50),curLifespan));
}

void run(int w, PVector d, int t, int h) {
Iterator<Particle> it = particles.iterator();
while (it.hasNext()) {
Particle p = it.next();
//p.run(w,d,t,h);
p.run(wo);
if (p.isDead()) {
it.remove();
}
}
}
}

class WeatherObject
{
String city;
float wind;
PVector dir;
float temp;
float hum;
color col;

void WeatherObject()
{

}
}

class XbeeMultiLightControl implements Runnable
{

int N = 0;
int NE = 1;
int E = 2;
int SE = 3;
int S = 4;
int SW = 5;
int W = 6;
int NW = 7;

int rVal = 100;
int gVal = 0;
int bVal = 0;

int fadeSpeed = 50;
int pulse = 10;

int[] payload;

boolean isRunning;

XBeeAddress64[] addr;

XBee xbee;

void XbeeMultiLightControl() {

}

void init()
{
println(“////////////////////////////////////”);

addr = new XBeeAddress64[4];
addr[0] = new XBeeAddress64(“00 13 a2 00 40 98 99 85”);
addr[1] = new XBeeAddress64(“00 13 a2 00 40 98 99 8c”);
addr[2] = new XBeeAddress64(“00 13 a2 00 40 98 99 83”);
addr[3] = new XBeeAddress64(“00 13 a2 00 40 92 d7 c7”);

xbee = new XBee();

try {
xbee.open(Serial.list()[0], 9600);
}  catch (Exception e) {
System.out.println(“XBee failed to initialize”);
e.printStackTrace();
System.exit(1);
}

}

void pulsate(int dir, int rv, int gv, int bv,int speed, int pulse)
{
fadeSpeed = speed;

payload = new int[] {rv, gv, bv, speed, pulse};

}

void run()
{
if (!isRunning)
{
//isRunning = true;
sendValue(addr[0], payload);
delay(fadeSpeed);
sendValue(addr[1], payload);
delay(fadeSpeed);
sendValue(addr[2], payload);
delay(fadeSpeed);
sendValue(addr[3], payload);

println(” thread is done!”);
xbeeThread.interrupt();
//isRunning = false;
}
}

void sendValue(XBeeAddress64 addr64, int[] payload)
{

ZNetTxRequest tx = new ZNetTxRequest(addr64, payload);

try{

ZNetTxStatusResponse status = (ZNetTxStatusResponse) xbee.sendSynchronous(tx, 500);
if (status.isSuccess()) {

}else{

}
} catch (XBeeException e) {
System.out.println(“NOOOO”+”\n”);

}}
}
And here is how it looked:
SETUP 
Demi created the Xbee setup which consisted initially of 4 breadboards with 2 LED’s each connected to 4 individual Xbees drawing from 4 rotating weather APIs. Lights were programmed to change colours according to the colour of the weather data being displayed on screen, which corresponded to weather API’s from 4 different cities (Tehran, Toronto, Bersheeba, Saskatoon). This weather data on screen was represented by a continuous loop of interconnected floating triangles, again, colour corresponding to individual city weather data, and direction and speed corresponding to wind direction and velocity from the weather APIs. Ryan and I toyed with a variety of light setups, and finally settled on using cubes with draft paper and cardboard as a base.
CONTEXT + FUTURE DIRECTIONS
There are many situations where we feel the Weatherbe could be a valuable installation, both indoors and outdoors. Whether displayed at an intersection on north, south, east and west corners of University ave, at an airport or a train terminal, there are many reasons you may be interested in “seeing” the weather. Maybe you’re in a controlled environment, like a car or an office building, and you want to know what it feels like outside. Maybe you’re waiting to board a plane and you want a quick overview of the current weather in the city you’re traveling to relative to where you are now. Or maybe you’re interested in having a smaller version of the installation in your home, representing cities that mean something to you, or to people you care about.
We think Weatherbe presents an efficient, vivid and beautiful way to experience the weather. It’s a unique way to visualize how the weather feels in various cities simultaneously, providing an accurate scope of various pieces of weather data with one simple display technique.

CONNECT 4000

CONNECT 4000 is a modern interpretation of the classic game Connect Four. In this fun new game, players make their decisions online and the results play out at a location in the real world.

Log on to www.connect4000.com with a friend and pick a colour. Select a column in which to drop your marble. But choose quickly: if the timer on the game board runs out, a column will randomly be chosen for you. The first player to line up four of their marbles is the winner. Hit the RESET button to clear the game board and play again!

Continue reading CONNECT 4000

Project 3: onemile

onemile

by Yuxi, Hudson, and Maz.

Concept: onemile

The goal of this project is to create a connection between individuals revolving around sharing past experiences. The system uses a hood, which records the experiences of the wearer (light, sounds, and step) at a random time when the hood is worn down. This recording is transmitted to a base station whenever the hood comes into proximity, triggering data visualization of that recording. A previous recording, from another hood wearer, is then transferred back down to the hood at this point, creating a network effect of shared experiences. The downloaded experience is then played back by the act of wearing the hood up, connecting two individuals across time, location and experiences.

Experience recording (Data 1)

Single user experience with data transmission, sending (data 1) and receiving (data 2)

Following the path of a single recorded experience (data 1) over time between two hoods


Playback of sent experience (data 1)

Video

 

Theory

The Object: Hood

Wearing hoods can serve many purposes, weather protection being foremost. It also serves other purposes, such as providing social cues. Hoods can communicate that a person do not want to connect with other people since: they create a personal space which only the hood wear occupies.

We’ve flipped these perceptions and experience around. Instead of withdrawing into a hood it becomes a social experience that is tapped into: two people occupy the same hood space. Pulling up the hood acts to remove the wearer from the situation they’re currently in and project him/her into a new space created by the sense experience of another hood wearer.

Experience Interpretation

A tweet, post or a profile picture allows an individual to project a self-image that they choose for their social network to interact with and experience. We want to challenge this: what would the interaction with and experience of another person be without knowing their name, demographic, or likeness; all the things we’re used to having, to identify that person? How would this experience be interpreted by each user?

As this is an interpretive experience, we don’t believe this to be a more “real” or “authentic” means of conveying a person or experience. What we are providing is a mean of experiencing another in a unique, unfamiliar method. New experiences have the power to open our eyes to other possibilities; perhaps onemile can provide such opportunities.

Technology

When worn down, our Arduino-equipped hood randomly initiates recording of the hood wearer’s experience. This recording consists of the light values, via photo sensor, steps taken, via accelerometer, and sound values, via microphone. The hood wearer is alerted that recording is taking place by the rhythmic toggling of a fan, near the hood wearer’s neck, to simulate breathing. This creates the illusion of another person closely following the hood wearer; which, in turn, acts as a reminder that their current experience will eventually be occupied by another person.

Our hood is equipped with an Xbee for wireless communication of stored data. When a hood wearer approaches our data visualization base station its Xbee is used to transmit the hood’s stored data to the base station’s Xbee. At this point, the Processing sketch running on our base station triggers data visualization of that recording. In addition to viewing their own data visualizations, users are presented with averages of other, past hood wearers’, data. This allows for users to connect on deeper level with their own experience by quantifying it along with other users’ experiences.

After a hood’s recorded data is transmitted to a base station a previous recording, from another hood wearer, is then transferred back down to the hood and any existing recordings are deleted/overwritten. When the hood is worn up this shared recording from another hood wearer is played back: light values via an LED strip around the hood edge, steps taken via a vibration motor at the chest, and sound values via speakers at ear level.

Development

Ideation

We began this project with three wholly different directions: linking two place digitally and letting inputs filter across, wearable technologies such as transformable clothing, and creating hardware that acts as driver for a new form of social network. Brainstorming sessions acted to distill out the idea that resonated the most with each team member creating shared experiences through wearable technology.

Developing this idea began by first understanding what the experience would be like to use our project as well as why one would use it. Discussion revolved around individuals that may currently be having particularly bad experiences. Our thought was that perhaps we could offer a means of escapism; we would offer a wearable device that would project oneself into another persons’ experience, a better experience. We also discussed the possibility of the opposite being true. Perhaps individuals having a bad experience could be comforted by the presence of people piggybacking off their experience and offering some form of solace. This would have taken the form of a simulated hug.

The form factor of the hood came out of discussions of devices and clothings that acts to remove people from the present space, in particular, reflections of people wearing hoods and headphones on public transit. In our interpretation, these individuals seem detached from the reality around them.

We also had to take stock of what inputs and outputs could be shared. We explored sound, light, breath, hug, pulse, vibration, GPS coordinates, and social media feeds. This quickly lead us to our final three inputs/outputs as cost, complexity and feasibility were weighted. Transmitting a hug was far too complex and potentially heavy, GPS data was costly and, like social media feeds, the form GPS output would take was unclear.

Tasks and Roles

Development on the hood generally broke down into three main categories and three sub categories. The main categories consisted of the hood, hood code, and visualization code. The sub categories consisted of the hood hardware, data, and branding. Each team member was charged with leading one main category while also contributing to the two related sub categories. The following diagram illustrates the categories, their relationships, and the team members charged to them:

Flow of ideas and work between Yuxi, Hudson, and Maz

Hardware / Software – Prototype Unit
Hudson

Prototype unit
Initial development focused primarily around the hardware and developing a prototype of the sensors which would eventually find their way into the hood. This prototype guided decisions made in other aspects of the project. For instance, it was from this prototype that it was discovered we needed space for two battery packs in the hood (see power limitations). There were cases, however, where external influences affected the layout of the prototype; our initial idea to use a flex sensor to detect the state of the hood proved impractical in practice, so the prototype design was changed to a switch.

Power limitations
The Arduino can provide power out in three manors, 5V at 40 mA from each I/O pin, 3.3V at 50 mA from the dedicated 3.3V pin, and a dedicated 5V safely up to about 500 mA. The devices underpinning our hood require widely varying levels of power and amperage. Powering directly off the I/O pins, 3.3V pin and using the 5V in combination with TIP120s to regulate the power flow seemed to be the power solution we required. Sadly, I found that drawing current from the 5V pin with our fan to simulate breathing would cause our sensor data, acceleration and frequency, to fluctuate, thus making data logging of any high precision to be impossible. A low-pass filter to regulate the voltage was a likely solution but my implementation of one seemed to limit all sensor variation altogether. The only present solution was to power the fan off a separate power supply despite the added weight.

Prototyping / development unit

Fritzing Diagram of the hood’s circuitry

Storing and Recalling Data

Arduino code flow / if statement decision tree.

Recorded experience data is saved onto the internal memory of the Arduino inside of arrays, with one array for each of the following: step time, light change time, light change value, sound change time, and sound change value. The two different arrays for both light and sound are linked via a shared counter value used for incrementing through both recording and replaying. Step also has a counter used to increment through step times, but no associated arrays such as intensity of step.

The data in these arrays are what is transferred whenever there is communication between our Arduino equipped Hood and our data visualizing and handling Processing sketch. A extra byte of data is transmitted first to identify the data type, then the array is incremented and transmitted with a comma delimiter.

To Arduino and Processing, this data looks like…

1,0,11
2,0,5,128
3,0,5,0
1,1,944
2,1,870,127
3,1,956,1069
1,2,1815
2,2,1763,125
3,3,1208,0

A stylized, human readable (used for debugging), version of that same data…

Step # : 0 , at time: 11
Light # : 0 , at time: 5 , with brightness: 128
Sound # : 0 , at time: 5 , with frequency: 0
Step # : 1 , at time: 944
Light # : 1 , at time: 870 , with brightness: 127
Sound # : 1 , at time: 956 , with frequency: 1069
Step # : 2 , at time: 1815
Light # : 2 , at time: 1763 , with brightness: 125
Sound # : 2 , at time: 1208 , with frequency: 0

Memory limitations
Using this system of array, memory capacity was an ever-present concern. Recording anymore than 50 data points per sensor was a surefire method to crash our Arduino Uno test system. This was because the hood is wholly reliant on the built in memory of the Arduino. Suffice to say, that does not provide much space for data storage. Time constraints prevented any deep exploration into using an SD-card reader to expand the memory. In order to be functional without the expanded memory provided by the SD-card I had to be very strict about how often data was logged, else run the risk of running out of space in seconds.

Hardware – The Hood
Yuxi


Prototype hood

Pattern, Fabric, Sewing and adding details
Concept- providing a private mental space anywhere
Solution: large sized hood, cover eyes, thick but soft fabrics.

Technical requirements – speakers, fan and vibration motor have to be placed on the right places in order to guarantee the experience.
Solution: nylon fastener tape, snaps and pockets boxes made by fabric.


Hood component pockets

Design requirements – To hide the wires
Solution: two layers, covers and pockets.


Hood sensor layout sketch


Hood circuitry and control system (Arduino)

Hooking up circuits
Due to lack of experience, we decided to use a breadboard, which caused us a lot of troubles at first. We later realized that it could have been better to solder everything on a protoboard in the first place. We decided to make the change on the day of the presentation. Unfortunately, we did not have enough time to test everything. The system broke before the presentation. It was a hard lesson for the team. I realized that it is very important to fully understand the specific features of differences circuit board and parts before implementing them on the project, as well as to test the possibilities in wearable gadgets. Therefore, in future projects, we are able to choose the appropriate parts and minimize the chance of malfunctions.


Hood breadboard circuit – initial sensors’ connection to Arduino

Hood protoboard circuit – final sensors’ connection to arduino

Up/Down detecting sensor
In order to realize our concept, which was hood down – recording, hood up – replaying, we needed to figure out the appropriate sensors to detect the hood movements. We tested number of sensors including tilt sensor, flex sensor, force sensor, magnet, snaps and the homemade sensor (nuts). Although all of them worked, we were still not satisfied because the fabric could not hold them very well. In the end we decided to use claps, which was not as good as the other sensors tested, but it worked consistently.

Alteration
1.Hood cannot be worn fully put down, because the Arduino was right above the fan. When the hood was worn down, the fabric cannot be bended because of the Arduino.

Solution: I switched the Arduino to the place where the battery pack was held (moved the battery pack somewhere else) and the problem was solved.

2. Due to the technical requirement, we had to add one more battery pack. The problem was if I put two battery packs at the back of the hood, it would be over weight and not justifiable.
Solution: I used the same extra fabric to make two straps. I placed one battery pack at the bottom of each strap in order to hang them in front of user’s body. That was to balance the weight between the front and the backside of the hood.

3. The most difficult thing was to always keep the fan facing the back of the hood wearer’s neck. The fabric was too soft to keep the fan at this exact spot.
Solution: I used the extra two straps (mentioned earlier for the battery packs) to cross over the fan area. When wearing the hood, the weight of the battery packs stretches the straps, which would in turn pulling the back of the hood up while holding the fan at the right place. In addition, I created a “scarf” at the bottom of the hood, which can bond around the user’s shoulder and steady the entire hood.

Hood internal layer (inside out) and fan with cover

Reflection from the project
From this project I realized that making clothes for wearable technology is tricky. Sometimes the concept of the project could influence the design greatly. However, I really enjoyed the process and I loved to solve the challenges through better design solutions. I have strong passion towards wearable technology, not only because I enjoy making clothes, but also to create things that are more practical to use. For example, to work out the conflicts between design and technical needs is my favorite part of the process. Starting from this project, I want to pay more attention to the current situation of the wearable technology, as well as the future trend of this industry.

Inside of the hood

Outside of the hood

Sensor locations on hood

Software – Visualization
Maz


Data visualization design, hand sketches

As a general rule, we opted for iconic visuals to represent the input data of the user. A frame of influence was webicons, software navigation icons and video game symbols. We thought to include a pulsing heart or lung in the corner of the data viz display merely to reflect the “breathing fan” embedded in the hood but the visualization got a bit cluttered so it was excluded. Obvious choices for the sound input were speakers, EQ waves, but finally a speaker icon that emitted pulsing waves was chosen due to its simplicity and universal recognition. A light bulb was chosen to represent the light input for similar reasons as the speaker icon. Lastly, for the movement input, we played around with a few whimsical angles such as GIFs of the Beatles walking around Abbey Road, and the stop hand and walking man found on street lights. Eventually, we all fell in love with a simple walking man GIF that also match the styles of the other two icons. Consistency in design choices was key to maintaining the aesthetic quality of the visualization.

Final Processing data visualization software

I received tremendous technical support from Hudson regarding the engineering of the code and how the data viz receives input from the Arduino and how the average data sets for each user experience switched upon the commanded of the viewer. The Processing library GifAnimation provided some support to integrate the GIF files into the display and from that I learned to adapt them for our specific requirements.

It was important for me to make sure that the data viz display was very much a part of the onemile experience and not simply something patched on. To address this, I kept the branding of the hood with the data viz display consistent and unified. This involve continual coordinating of colour, shape and texture with Yuxi.

Additionally, I wished to engage the hood wearer with the visualizations of their data beyond just looking at it. My intent was to have them engage directly with the data. Specifically, regions of the display would respond dynamically when moused over them, providing additional information about their data in a look not unlike the rest of the visualization so as not to be too jarring.

Future Improvements

Memory

Ideally we would augment the memory of the Arduino with an SD-card reader. Time constraints prevented this from being an option for this iteration. Additionally, our current data storage and recall methods, both on the hood and the visualization system, limit our ability to store large amounts of data as well as recall any data after power-off. We explored the possibility of storing data in a text file on the computer for later recall but again found time to be a limited factor.

Sensors

Our team has explored the possibility of recording light colour in addition the light intensity but find the hardware solutions available to use to be lacking. The distance from which colour could be perceived by the colour sensor was limited to only a few centimetres. As such, we chose not to implement a colour sensor in this project, though the thought of providing such data is still appealing and worth further investigation.

Sound detection is another area where hardware proved to be a limiting factor. The microphone in our current setup reports frequency but fails to report amplitude. As a result, we are not able to get a true reading of loudness around our hood as we had intended. Instead we are only able to report on sound activity (changes in frequency).

Control Unit and Power

The hood is very heavy at the moment.  This partly a result of the two power supplies and also because of the Arduino we used.

Regarding the batteries, initially our design had called for a single power supply to power the Arduino which, in turn, would power the rest of the hood.  We had to scrap this due to interference created in our sensor data by the fan powering on and off. A low-pass filter to regulate the voltage is an ideal solution and we’d like to explore this more.

Similarly, our choice of Arduino was not our first selection. Our Arduino Uno test system, a unit which has 32KB of built in memory, would lock up from lack of memory if recording more than 50 sensor events. For comparison, the Arduino LilyPad, a better Arduino for wearables, has only 16KB of memory, only 14KB of which is accessible. Our sketch alone is 13KB. The Arduino LilyPad, at least in our project’s current configuration, does not work.

Data Transmission

Unlike our original concept, the hood doesn’t transmit or playback data in real-time. Our concept had originally been to transmit data in real-time over wifi networks in a high bit-rate stream but there did not appear to be much precedent in this area. As such we scaled back to a better documented technological solution: using Xbees to transmite recorded data logs. Ideally, given enough time, we would explore and implement our initial concept with wifi data streaming.

Code

The following is a link to a Dropbox folder containing our code:
https://www.dropbox.com/sh/4gnaxwdszczew8h/9YbJ-Ula0W

Sources / Inspiration

Hood and Wearable Technology

Like-A-Hug

A vest that gives you a squeeze for every facebook “Like” you get
By Melissa Kit Chow
http://www.melissakitchow.com/Like-A-Hug

Fashion Trends

Scarf hood trend

As found on Google image search.

Photos here

 

Data Logging

We explored many devices that logged daily activity such as step count and sleep patterns. This research proved vital when we began to visualize our logged data. Of particular help was the method in which the Jawbone Up was used to log step data, as it cannot be graphed in the same mode as light or sound.

Jawbone UP

https://jawbone.com/up

Nike+ Fuel Band

http://www.nike.com/us/en_us/lp/nikeplusfuelband

Data Visualization

Felton

Nicholas Felton’s Personal Annual Reports that weave numerous measurements into a tapestry of graphs, maps and statistics that reflect the year’s activities.
http://feltron.com/ar11_02.html

HEARTSTRINGS

We began this project by brainstorming ideas to on how to encompass our interests, address a problem in our lives, and connect people who may not necessarily be in the same space.

This led us to the conclusion that enhancing/augmenting the nature of webcam conversations was the direction we were aiming to go. Since a lot of us in the program aren’t from Toronto, Skype conversations with the significant other become routine.

The question, now formed, was how can we make Skype conversations more intimate? After researching online, we came across an academic paper entitled ‘Intimate Heartbeats: Opportunities for Affective Communication Technology’ by Joris Janssen et. al. The findings of this study concluded that “using self-report and behavioral tracking in an immersive virtual environment, that heartbeat perception influences social behavior in a similar manner as traditional intimate signals such as gaze and interpersonal distance”.

From this, we decided that effectively communicating heartbeat information to your Skype partner would be the ideal way to increase the feelings of intimacy when conversing with one’s significant other. We also found projects which we used as inspiration, and also as a benchmark for what avenues not to pursue (kissenger… a little creepy).

After picking up a Pulse Heart Sensor, we began working on some initial prototypes – a simple brooch that one would attach to their shirt, with an accompanying LED that would pulse to the beat of one’s heart.

However, as stated in the article we read and as common sense would have it, “the stimulus must be attributed to the conversational partner in order to have influence”, and as such, an LED blinking doesnt really do the job on its own.

So we moved on and attempted to work with multiple inputs/outputs including a vibe motor, heating pad, touch sensors, and RGB leds.

Due to the power requirements of the heating pad, vibe motor and rgb led, and the power constraints of the arduino, we had to limit ourselves to the input of the heart beat information supplied by the pulse sensor and a stretch sensor to control your mood, and a few outputs including an RGB LED and a vibe motor for a tactile sensation.

From our initial brooch prototype, we tossed around some ideas and came to the conclusion that a wearable solution would be ideal – it would hide all the electronics nicely and also allow for the pulse sensor to be within an ear’s distance.

For the aesthetic, we wanted the prototypes to feel comforting, and have an organic feel, so that it wouldn’t be cumbersome and feel like an extension of oneself “connecting” to their partner through technology.

And this required us (with some help from Borxu & Yushi) to learn how to use a sewing machine.

 

http://youtu.be/D-8PpsDpong

We used code from the Pulse website as well as Cosm to send heartbeat information over the internet and receive it in arduino, such that the vibe motor and led one was wearing would react to the heartbeat of their partner. (Arduino Code | Processing Sketch)


The final iteration was sewn onto a red velour sweater, for that touch of class, and was made using a Lilypad, some conductive thread, and was soldered onto a perf. board.

We also used a stretch sensor which would change the color of the heartbeat LED from blue to red, however that didnt make the final iteration due to a broken sensor (however can be seen in our sweet video).

After some debating, we decided that some sort of data visualization was needed in order to bring it all together, so with a basic processing sketch, we were able to map the two heartbeats, and as they became closer in unison, so to did the circles spatially.

Going forward, we would develop our second prototype into a similar sweater-sewn iteration and include more outputs, such as heat and sound.

Moreover, since there is a heavy dependence on Skype, we imagine developing an API to integrate the data visualization into the Skype window so that it is not competing with the conversation window for your attention. The processing sketch itself would also have to be developed extensively so that the aesthetic and efficacy of it are markedly improved.

For this particular project, we wanted to have the arduino wired so that the organic feel of connecting the usb which is protruding as the sweater’s umbilical cord was emphasized. However in a future rendition, a wireless version would be a good option to make the Heartstrings sweater more mobile.

Eschatology 1/4 – Lindy, Yifat, Alex, Torin

For full documentation go to: http://eschatology14.wordpress.com