The Lightness of Sound

By Jeremy Littler and Mitzi Martinez

What is The Lightness of Sound?

SEE The Lightness of sound:

 

Based in the LUMARCA design created by Albert Hwang, Matt Parker and Eliot Woods (http://lumarca.info/), The Lightness of Sound is an installation designed to visualize music in a 3D light display, adding a new dimension to the interaction of the musicians with their performance, by adding a visual element displayed in a physical object that reacts on every note that is being played.

 For the musicians, this completes the sensorial experience as by playing they are physically touching the instrument to produce notes, they hear it, and they see the volumetric display in the WireMap reacting to their input by showing changes of length and color of the light.

 For the spectators, it also adds a new dimension as they can see the synergy between music, instrument and musician condensed in a physical object that displays this interactions and the flow of the music in a very visual and almost tactile way.

How it works?

On the technical side, the Lightness of Sound project goal, was to integrate a wireless MIDI system (in this case a MIDI guitar) with a projected visualization. Ideally, live music could be performed that seamlessly blended both the visual and auditory elements into a seamless experience for the audience.  In theory, the wireless MIDI system would allow musicians to create a dynamic musical compositions influenced the projected content, while at the same time being influenced musically by visualizations.  The core elements of the wireless MIDI system were as follows:

1)   A Midi Guitar with support for battery based operation.

2)   An Arduino with MIDI input capabilities.

3)   An XBee based transmission system (for MIDI).

4)   A Processing application that integrated audio and projection playback functions.

5)   A virtual MIDI system that enabled Processing to convert MIDI notes to sound.

MIDI OUTPUT (FOR INFORMATIONAL PURPOSES)In terms of the hardware/software requirements Arduinos are easily configured for MIDI output  (See example below).

 

 The hardware requirements are a MIDI Connector  (PRT-09536 PRT-09536 from Sparkfun), three segments of wire and a 220 ohm resistor.  To output MIDI, you can install the RWMidi Library for processing or MidiBusMIDI content generated by Processing can be patched via Apple’s IAC Driver to Garage Band (or other sound generators) or connected to a USB-Midi interface (e.g., the Uno) and then taken as an input in the sound generator. There is an example of this approach at Instructables. The process of configuring an Arduino for MIDI input, as was required by this project, is more complex…

MIDI INPUT

Developing a solution based on MIDI input is more complex than MIDI output. There are essentially two ways to configure an Arduino as a MIDI input device. The least costly option is to breadboard a MIDI IN interface (see below) using basic electronic components.

Source: Notes and Volts:

http://www.notesandvolts.com/2012/01/fun-with-arduino-midi-input-basics.html

 

The parts requirements for this project are:

 

6N138 Optocoupler

A 270 ohm resistor

A 220 ohm resistor

A 1N914 Diode

A Midi Socket

3 wires.

The total cost of these items is under $10.00.

Note: If connected, this circuit will conflict with the programming of the Arduino. Therefore, the circuit must be disconnected when performing this task. It would be worth installing a “programming” switch to simplify development.

Despite the circuit being fairly simple, Jeremy spent an entire day attempting to get it to work (see below) without success. Simply put, the circuit would not communicate MIDI information to the Arduino. Perhaps others will have more success with it. It’s worth noting that there were comments online that others were unable to get this to function. It’s possible that changes to the Arduino environment or hardware have rendered this circuitry inoperative.

The second method (as used in this project) of enabling MIDI IN on an Arduino is to attach the SparkFun MIDI Shield to the Arduino (PRT-09595 from SparkFun). However, it’s worth noting that this module (which handily includes a programming/running switch) does block access to the RX/TX headers on the Arduino. This presents a challenge when combining MIDI Input with an XBee via an Arduino.

To receive notes from the MIDI shield an Arduino library is required. Specifically, the Arduino Midi Library (version 3.2+).  As the MIDI shield overrides the serial communications sub-system on the Arduino it is not possible to simultaneously use a MIDI shield and an XBee for serial communication using standard serial techniques. To overcome this limitation, the Arduino can be configured to communicate with the XBee using the SoftWare Serial Library.  However, this does limit the baud rate of the XBee communication to a maximum of 9600bps.

Source: SparkFun


The Arduino code for the MIDI IN project is as follows:

//*******************

// Wireless Midi to Xbee

// By Jeremy Littler

// Using 2 Key Libraries. See below

// Note: on the Yamaha - EZ-AG the Note Range it is 41-76

// ****************

//

//                            ==================

//   ooo                   ===================

//   ===               ===============

//   ====================[]===[]=====   ROCK ON!

//   ===              ================

//   ooo                 ==================

//                             ===============

// THE ESSENTIAL

// SoftSerial Library from Arduino

// http://arduino.cc/en/Reference/SoftwareSerial

// THE ESSENTIAL

// Arduino Midi Library from Ardiono

//http://www.arduino.cc/playground/Main/MIDILibrary

// fortyseveneffects (at) gmail (dot) com.

// To overcome the Sparkfun midi adapter taking

// Over the Flipping Physical Serial Port. use SoftwareSerial

// to communicate with the XBee

//Don't Forget to That the Midi Shield Needs to

// Be Manually Switched (near the midi plugs)

// Between Program and Operation Mode

#include <SoftwareSerial.h>

#include <MIDI.h>  // Add Midi Library

// LED is defined to determine (easily) the MIDI Value

#define LED 13    // Arduino Board LED is on Pin 13

// Use Software Serial to talk to the Xbee. Whoppeeeee!

// This used Pin 11 to Output the Midi Value

// Connect the Jumper Wire from

// From Pin 11 on the Arduino to the

// XBee Station

SoftwareSerial myXbeeSerial(10, 11); // RX, TX

String MidiNoteString = "02#";

//  Channel (Ignored), Pitch (Kept), and Velocity (Ignored)

void MyHandleNoteOn(byte channel, byte pitch, byte velocity) {

// Have Received Data so Prepare it to be printed to the XBEE

String MidiNoteString = "02#" + String(pitch);

if (velocity == 0) {//A NOTE ON message with a velocity = Zero is actualy a NOTE OFF

digitalWrite(LED,LOW); //Turn LED off

// THE 01 VALUE DEMARKS MIDI DEVICE 1

// THE 02 VALUE DEMARKS MIDI DEVICE 2

String MidiNoteString = "02#" + String("999");

myXbeeSerial.println(MidiNoteString);

} else {

digitalWrite(LED,HIGH);//Turn LED off

// THE 01 VALUE DEMARKS MIDI DEVICE 1

// THE 02 VALUE DEMARKS MIDI DEVICE 2

myXbeeSerial.println(MidiNoteString);

}

}

void setup() {

myXbeeSerial.begin(9600);

pinMode (LED, OUTPUT); // Set Arduino board pin 13 to output

// OMNI sets it to listen to all channels.. MIDI.begin(2) would set i

MIDI.begin(MIDI_CHANNEL_OMNI);  // Initialize the Midi Library.

// Kill Thru Mode as it hogs performance

MIDI.turnThruOff();

// Don't Need Sysex Message so turn that off

MIDI.disconnectCallbackFromType(SystemExclusive);

//Register a Function to Call Note on/off Messages

MIDI.setHandleNoteOn(MyHandleNoteOn);

}

void loop() { // Main loop

MIDI.read(); // Continually check what Midi Commands have been received.

//myXbeeSerial.println("Hello, world?");

}

XBEE CONFIGURATION

 Two XBee units are configured using CoolTerm (in OS/X) with the following settings:

XBee Receiver:

Pan ID: 2171 - ATID2171 (XBee 1 and 2 must have the same Pan ID).

ATDL 0

ATMY: 1

XB (MIDI) Sender:

Pan ID: 2171 - ATID2171

ATDL: 1

ATMY: 0

Both Xbee units should be set to a data rate of 9600 BPS (ATBD3 in CoolTerm)

 One XBee is wired to the headers on the SparkFun MIDI shield, as follows:

  • Pin 11 to Pin 3 on the XBee
  • +5V to Pin 1 on the XBee
  • GND to GND on the XBee

We worked over the Processing Sketch created by Matt Parker, changing the display and combining the MIDI Read Functionality and the Project visualization that we used is the following:

 

/*

Wiremap Renderer for 2 Globes
by

/ \
______/________\______
/ \ / \
/________\ /________\______
/ \ | \
/ / \ \_____ | \
/________\______ | /
\ / \ |_____/
\ /________\
\ \ /
\_____ \ /
/ \
/ \

For more information on the project please visit:
http://wiremap.phedhex.com

This program builds two separate 3d globes. I have two separate functions (& sets of variables) because I haven't quite yet figured out how call a function twice. Elementary stuff, I know, but I'll get to it when I can.

Some conventions to be aware of:

1 - This particular program builds in millimeters.
2 - Also, I use a left-handed coordinate system, with positive X going right, positive Y going up, and positive Z going forward.
*/

// Fullscreen stuff:

/* Variable declarations
---------------------------------------------------------*/

/* Physical Wiremap, in inches
---------------------------------------------------------*/

float depth = 70.0; // The mapline lies 3 meters away from the projector's focal point
float map_length = 32.0; // The mapline is 1.28 meters wide
float depth_unit = 0.500; // Each depth unit is 5 millimeters
float map_unit = 0.7; // Each mapline unit is 5 millimeters
int wire = 64; // There are 128 wires in this Wiremap
float depth_thickness = 30.0; // How deep is the field (perpendicular to the mapline)
/* Projector
---------------------------------------------------------*/

float ppi = 40; // Pixels per millimeter (unit conversion). Only true for mapline plane - 4 pixels every 5 millimeters
int string_pix_count = 9; // How many columns of pixels are being projected on each string //Mitzi's comments: here I changed the thickness of the lines.
/* Map
---------------------------------------------------------*/

float[] map = new float[wire]; // example: map[0] = 90 means that the first string is 45 cm away from the mapline
float[] x_by_ind = new float[wire]; // x coordinate for each wire
float[] z_by_ind = new float[wire]; // z coordinate for each wire
/* Globe A
---------------------------------------------------------*/

float[] globe = new float[3]; // globe x,y,z coords
float radius = 31.00; // default globe radius
int dot_height = 15; // height of surface pixels.
boolean render_globe = true; // toggle globe rendering

/* Key input
---------------------------------------------------------*/

float step = .2; // how far the globe moves / button press
boolean mouse = true; // is mouse clicked?
int colorval_r = 200; // red
int colorval_g = 100; // green
int colorval_b = 255; // blue
boolean xpin = false; // the mouse controls the globe's y & z axis
boolean ypin = true; // x & z
boolean zpin = false; // x & y
int start_time = 0; // for beat mapper
int end_time = 0; //
float beat_multiplier = 1; // multiplies freqeuncy of how often beat hits

 

/* Beat Mapper Variables
---------------------------------------------------------*/

int[] last_32 = new int[32]; // last 32 times the spacebar has been pressed
int times_struck; // nubmer of times spacebar struck since timer was reset
int first_strike; // millis value for when timer was reset
int period = 500; // time between beats (this is the metronome)
int offset = 1; // how far away in time we are from the last beat
/* wave variables
---------------------------------------------------------*/
int trail = 350; // number of iterations of past mouse clicks we keep
int[] click_time = new int[trail]; // array of times (millis) associated w/ clicks
int[] click_x = new int[trail]; // array of x locales for clicks
int[] click_y = new int[trail]; // array of y locales for clicks
float[] click_x_trans = new float[trail]; // translations from mouse x to xyz
float[] click_y_trans = new float[trail]; // translations from mouse y to xyz
float amplitude = .6; // amplitude of waves
int decay = 3000; // how long it takes for waves to die down
float wave_velocity = .035; // inches/milliseconds
int trail_frequency = 10; // milliseconds - NOT frequency of wave, but how often a new value gets pushed into the trail arrays (above)
int trail_cycle_count; // this gets updated once every (trail_frequency)
int trail_cycle_count_compare; // this is used to check to see if we need a new value
int water_color = 0; //

float plane_angle = .0; // the angle of the plane of the water (think m in y = mx + b)
float plane_intercept = 0; // where the plane intersects the origin (think b in y = mx + b)
import processing.opengl.*;
// *******
// JL CODE
// ******

// ********************
// CODE TO READ SERIAL PORT FROM XBEE
// AND CONVERT TO VIRTUAL MIDI
// **************** *
import rwmidi.*;
MidiOutput output;
import processing.serial.*;
Serial myPort; // Create object from Serial class
String val ="0"; // Data received from the serial port
String CurrentNote = "0"; // Current Note in the Serial Buffer
String LastNotePlayer = "0"; // Last Note Played
int PitchRanged = 0; // Converts the Midi Note Value to a More Predicatable Range Value

String inBuffer = ""; // Serial Port Value Received
String postinBuffer; // Used to Time WhiteSpace entc
String line;
// Used to Control the Color of the Tips in the Project Window
int tipR = 200;
int tipG = 70;
int tipB = 205;
float tipvalue=0;
int GlobeTip = 255;

// Controls the Globe Movement Accross the X Axis
float midi_mousex = 0;
// Controls the Globe Movement Accross the Y Axis
float midi_mousey = 0;
// Total Notes Played. This creates an iterative environment
int totalnotesplayed = 0;
// Controls
float midi_mouse_glide_x = 0;
float midi_mouse_glide_y = 0;

// END JL CODE *******

 

 

static public void main(String args[]) {
PApplet.main(new String[] { "--present", "olasverdes" });
}

void setup() {

// JL START CODE
// List The Serial Devices
println(Serial.list());

// Soft Serial can only send at 9600 safely. Therefore, the Xbee has
// Been Configured to 9600 Baud.
myPort = new Serial(this, Serial.list()[0], 9600);
myPort.bufferUntil('\n');
// Configures The System for Midi
// creates a connection to IAC as an output
output = RWMidi.getOutputDevices()[0].createOutput(); // talks to garageband
// END JL CODE

// JL - HAD TO REMOVE OPEN GL COMMENT
size(displayWidth, displayHeight);
//size(displayWidth, displayHeight, OPENGL);
background(255);
colorval_r = 25; //Mitzi's notes: color of the globe that floates over the wave
colorval_g = 200;//Mitzi's notes: color of the globe that floates over the wave
colorval_b = 100;//Mitzi's notes: color of the globe that floates over the wave
loader();
}

void draw() {
noCursor();
noStroke();
frameRate(30);
fill(0); //Mitzi's notes: this is the background color
rect(0, 0, width, height);

sineSurface();
}

void sineSurface() {

/* trail_frequency appends clicks to the mouse trail arrays */

int remainder = millis() % trail_frequency;
trail_cycle_count = (millis() - remainder) / trail_frequency;
if (trail_cycle_count != trail_cycle_count_compare) {
trail_cycle_count_compare = trail_cycle_count;

// JL Changed
append_click(int(midi_mouse_glide_x), int(midi_mouse_glide_y));

// append_click(mouseX, mouseY);
}

float[] time_since_click = new float[trail]; // the difference between now and the array of clicks
float[] amp_modifier = new float[trail]; // the amp according to decay and time since click
float[] distance_since_pass = new float[trail]; // the distance since the head of the wave has passed the string
float[] distance_since_pass_fraction = new float[trail]; // the distance gets multiplied by a fraction for beat mapping (period)
float[] time_since_pass = new float[trail]; // amount of time that has passed since head of wave & wire intersection
float[] wave_head_distance = new float[trail]; // distance between epicenter and head of wave
float[] amp = new float[trail]; // amplitude of wave @ wire point according to mouse movement & beatmapping

/* for each wire... */

for(int i=0; i<wire; i+=1) {
float final_amp = z_by_ind[i]*plane_angle + plane_intercept ; // the baseline for the final amplitude is an upward slope when looking @ the wiremap... used y = mx + b

for(int x = 0; x < trail; x ++ ) {

float local_hyp = sqrt(sq(x_by_ind[i] - click_x_trans[x])+sq(z_by_ind[i] - click_y_trans[x]));
time_since_click[x] = millis() - click_time[x];
wave_head_distance[x] = time_since_click[x] * wave_velocity;
distance_since_pass[x] = wave_head_distance[x] - local_hyp;
distance_since_pass_fraction[x] = distance_since_pass[x] / float(period / 6);
time_since_pass[x] = distance_since_pass[x] / wave_velocity;
if (time_since_pass[x] > 0 && time_since_pass[x] < decay ) {
amp_modifier[x] = time_since_pass[x] / decay - 1;
}
else {
amp_modifier[x] = 0;
}
amp[x] = - amplitude * amp_modifier[x] * sin((2 * PI * distance_since_pass_fraction[x]));

final_amp = final_amp + amp[x];
}

float y_top_coord = final_amp;
float y_bot_coord = -20;
float y_top_proj = y_top_coord * depth / z_by_ind[i]; // compensate for projection morphing IN INCHES
float y_bot_proj = y_bot_coord * depth / z_by_ind[i];
float y_height_proj = y_top_proj - y_bot_proj;
// JL Modified - Added Variables for TipR, TipG and TipB
fill(tipR,tipG,tipB); //Mitzi's notes: here you change the color of the tip in every line // draw a rectangle at that intersect

//int tipR = 200
//int tipG = 70
//int tipB = 205
// rect 1 is top dot for sliver
float left1 = i * (width) / wire;
float top1 = (height/ppi - y_top_proj) * ppi;
float wide1 = string_pix_count;
float tall1 = dot_height;
rect(left1, top1, string_pix_count, tall1); // draw a rectangle at that intersect

// rect 3 is filler for sliver
fill(200, 255, water_color); // Mitzi's notes: here you change the color of the wave.

float left3 = i * (width) / wire;
float top3 = (height/ppi - y_top_proj) * ppi + dot_height;
float wide3 = string_pix_count;
float tall3 = y_height_proj * ppi - (dot_height * 2);
rect(left3, top3, string_pix_count, tall3); // draw a rectangle at that intersect
}

// Jeremy Changed Code
float globe_x = (midi_mousex / float(width)) * (map_length) - (map_length / 2);
// float globe_x = (mouseX / float(width)) * (map_length) - (map_length / 2);
float globe_z = depth - (midi_mousey) / float(height) * (depth_thickness);
// float globe_z = depth - (mouseY) / float(height) * (depth_thickness);

float y_amp = globe_z*plane_angle + plane_intercept;
for(int x = 0; x < trail; x ++ ) {

float local_hyp = sqrt(sq(globe_x - click_x_trans[x])+sq(globe_z - click_y_trans[x]));
time_since_click[x] = millis() - click_time[x];
wave_head_distance[x] = time_since_click[x] * wave_velocity;

distance_since_pass[x] = wave_head_distance[x] - local_hyp;
distance_since_pass_fraction[x] = distance_since_pass[x] / float(period / 6);
time_since_pass[x] = distance_since_pass[x] / wave_velocity;
if (time_since_pass[x] > 0 && time_since_pass[x] < decay ) {
amp_modifier[x] = - time_since_pass[x] / decay + 1;
}
else {
amp_modifier[x] = 0;
}
amp[x] = - amplitude * amp_modifier[x] * sin((2 * PI * distance_since_pass_fraction[x]));

y_amp = y_amp + amp[x];
}
float globe_y = y_amp;
float radius = (sin(TWO_PI * float((millis() - offset) % period) / float(period)) + 1) / 2;
if(render_globe == true) {
gen_globe(globe_x, -globe_y, globe_z, 5);
}
//println(globe_x + " " + globe_y);
if (millis() > 10000) {
}
}
void append_click(int local_mouseX, int local_mouseY) {
click_time = subset(click_time, 1);
click_x = subset(click_x, 1);
click_y = subset(click_y, 1);
click_x_trans = subset(click_x_trans, 1);
click_y_trans = subset(click_y_trans, 1);
click_time = append(click_time, millis());
click_x = append(click_x, local_mouseX);
click_y = append(click_y, local_mouseY);
click_x_trans = append(click_x_trans, (local_mouseX / float(width)) * (map_length) - (map_length / 2));
click_y_trans = append(click_y_trans, depth - (local_mouseY) / float(height) * (depth_thickness));
}
void gen_globe(float x, float y, float z, float rad) {
for(int i = 0; i < wire; i += 1) {
if((x_by_ind[i] >= (x - rad)) && (x_by_ind[i] <= (x + rad))) { // if a wire's x coord is close enough to the globe's center
float local_hyp = sqrt(sq(x_by_ind[i] - x) + sq(z_by_ind[i] - z)); // find the distance from the wire to the globe's center
if(local_hyp <= rad) { // if the wire's xz coord is close enough to the globe's center
float y_abs = sqrt(sq(rad) - sq(local_hyp)); // find the height of the globe at that point
float y_top_coord = y + y_abs; // find the top & bottom coords
float y_bot_coord = y - y_abs; //
float y_top_proj = y_top_coord * depth / z_by_ind[i]; // compensate for projection morphing
float y_bot_proj = y_bot_coord * depth / z_by_ind[i];
float y_height_proj = y_top_proj - y_bot_proj;
/* Top dot
---------------------------------------------------------*/
fill(colorval_r, colorval_g, colorval_b); // Mitzi's notes: here you can change the color of the globe as well. Fill the globe pixels this color
float left1 = i * (width) / wire;
float top1 = (height/ppi - y_top_proj) * ppi + dot_height; // ppi = pixel / mm. These are conversions to & from pixels and mm
float wide1 = string_pix_count;
float tall1 = y_height_proj * ppi - (dot_height * 2);
rect(left1, top1, wide1, tall1);
fill(GlobeTip); // Mitzi's notes: here you change the color of the tip of the globe. If you put fill(255) it will be white.

/* Top Surface
---------------------------------------------------------*/
float left2 = i * (width) / wire;
float top2 = (height/ppi - y_top_proj) * ppi;
float wide2 = string_pix_count;
float tall2 = dot_height;
rect(left2, top2, wide2, tall2);

/* Bottom Surface
---------------------------------------------------------*/
float left3 = i * (width) / wire;
float top3 = (height/ppi - y_bot_proj) * ppi - dot_height;
float wide3 = string_pix_count;
float tall3 = dot_height;
rect(left3, top3, wide3, tall3);
}
}
}
}

 

void mousePressed() {
if (mouseButton == LEFT) {
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
append_click(mouseX, mouseY);
} else if (mouseButton == RIGHT) {
if (water_color == 255) {
water_color = 0;
} else {
water_color = 255;
}
}
}

void keyPressed() {

/* Globe A
---------------------------------------------------------*/
if (true == true) {
if (key == 'w') { // adds value to the dimension that the mouse cannot move in
if (xpin == true) {
globe[0] = globe[0] + step;
} else if (ypin == true) {
globe[1] = globe[1] + step;
} else if (zpin == true) {
globe[2] = globe[2] + step;
}
} else if (key == 's') {
if (xpin == true) { // subtracts value from the dimension that the mouse cannot move in
globe[0] = globe[0] - step;
} else if (ypin == true) {
globe[1] = globe[1] - step;
} else if (zpin == true) {
globe[2] = globe[2] - step;
}
} else if (key == 'e') { // adds to radius
radius = radius + step;
} else if (key == 'd') { // subs from to radius
radius = radius - step;
} else if (key == 'a') { // allows mouse control for radius (hold down 'a' and bring mouse up or down)
radius = (height - mouseY) * .8;
mouse = false;
} else if (key == 'q') { // stops ball in place so that you can pop it somewhere else
mouse = false;
} else if (key == 'z') { // color control (hold down buttons and bring mouse up or down)
colorval_r = (height - mouseY) * 255 / height;
} else if (key == 'x') {
colorval_g = (height - mouseY) * 255 / height;
} else if (key == 'c') {
colorval_b = (height - mouseY) * 255 / height;
} else if (key == 'v') {
colorval_r = (height - mouseY) * 255 / height;
colorval_g = (height - mouseY) * 255 / height;
colorval_b = (height - mouseY) * 255 / height;
} else if (key == '1') { // x y z pin switches
xpin = true;
ypin = false;
zpin = false;
} else if (key == '2') {
xpin = false;
ypin = true;
zpin = false;
} else if (key == '3') {
xpin = false;
ypin = false;
zpin = true;
} else if (key == 't') { // beat mapper buttons - start, stop, effects, and multipliers
start_time = millis();
} else if (key == 'y') {
end_time = millis();
period = end_time - start_time;
offset = start_time % period;
} else if (key == 'g') {
beat_multiplier = 1;
} else if (key == 'h') {
beat_multiplier = 2;
} else if (key == 'j') {
beat_multiplier = 4;
} else if (key == 'k') {
beat_multiplier = 8;
} else if (key == 'b') {
if (render_globe == false) {
render_globe = true;
} else {
render_globe = false;
}
}
}
}
void keyReleased()
{
if(mouse == false) {
mouse = true;
}
if(key==' ') {
if(millis()-last_32[31] > 1500) {
last_32[31] = 0;
}
last_32 = subset(last_32, 1);
last_32 = append(last_32, millis());
for(int i=31; i>=0; i--) {
if(last_32[i] == 0) {
times_struck = 31 - i;
first_strike = last_32[i+1];
break;
} else {
times_struck = 32;
first_strike = last_32[0];
}
}
if(times_struck > 1) {
period = (last_32[31] - first_strike) / (times_struck - 1);
}
offset = last_32[31];
}
}
void loader() { // loads data for this particular wiremap
map[0] = 15;
map[1] = 13;
map[2] = 0;
map[3] = 29;
map[4] = 37;
map[5] = 6;
map[6] = 31;
map[7] = 14;
map[8] = 9;
map[9] = 0;
map[10] = 12;
map[11] = 24;
map[12] = 3;
map[13] = 26;
map[14] = 39;
map[15] = 18;
map[16] = 3;
map[17] = 28;
map[18] = 11;
map[19] = 18;
map[20] = 1;
map[21] = 20;
map[22] = 24;
map[23] = 8;
map[24] = 7;
map[25] = 22;
map[26] = 17;
map[27] = 34;
map[28] = 37;
map[29] = 1;
map[30] = 23;
map[31] = 10;
map[32] = 2;
map[33] = 33;
map[34] = 6;
map[35] = 34;
map[36] = 27;
map[37] = 12;
map[38] = 19;
map[39] = 25;
map[40] = 11;
map[41] = 14;
map[42] = 5;
map[43] = 15;
map[44] = 27;
map[45] = 4;
map[46] = 25;
map[47] = 8;
map[48] = 32;
map[49] = 35;
map[50] = 7;
map[51] = 30;
map[52] = 21;
map[53] = 4;
map[54] = 16;
map[55] = 2;
map[56] = 20;
map[57] = 17;
map[58] = 38;
map[59] = 22;
map[60] = 32;
map[61] = 36;
map[62] = 30;
map[63] = 10;

for(int i = 0; i < trail; i ++ ) {
click_time[i] = 0 - (i * 500);
}
for(int j=0; j<wire; j++) { // calculate x and z coordinates of each wire
float xmap = (0 - (map_length / 2)) + j*map_unit;
float hyp = sqrt(sq(xmap) + sq(depth));
z_by_ind[j] = depth - map[j]*depth_unit;
x_by_ind[j] = xmap - xmap*map[j]/hyp*depth_unit;
}
}

 

//***********
// JL CODE TO GRAB MIDI INPUT
// FROM XBEE
//**************
void serialEvent (Serial myPort) {

try {

// The Midi Guitar - EZ AG I am using
// Produces Midi Note Values from a
// Total Range of
// Open E = 52
// High E on 6th String = 88
// You will need to adjust the values (i.e. using the range command possible).
// To reflect your particular midi device
// 1st (Low) 52 Open - 64
// 2nd 57 Open to 69
// 3rd 62 open to 74
// 4th 67 open to 79
// 5th 71 open to 83
// 6th 76 open to 88
inBuffer = myPort.readStringUntil('\n');
postinBuffer = trim(inBuffer);
CurrentNote = postinBuffer.substring(3,5);
//CurrentNote = trim(CurrentNote);
// send a note 200 mS long
int duration = 200;
int note = 0;
int channel = 1; // GB seems to ignore this, but mehtod needs it

// Definately Wrap this Code or there may be catastrophic exceptions of the serial port gets overloaded

try {

if (CurrentNote == null) {
return;
} else {

if (CurrentNote.equals("99")) {
int velocity = 0;
note = 0;
int success = output.sendNoteOn(channel, note, velocity); // sendNoteOn returns 0 = fail, 1 = success
delay(duration); // note length -see keys below for better solution
success = output.sendNoteOff(channel, note, velocity); // sendNoteOff returns 0 = fail, 1 = success
note = 0;

} else {

// This is protected by a try box to ensure a never lock situation

try {

note = int(CurrentNote); // midi notes are numbered 0 -127 -- not all notes are played by all voice
LastNotePlayer = (CurrentNote); // Store the last note in a variable
int velocity = 127; // range 0-127, 127= max
int success = output.sendNoteOn(channel, note, velocity); // sendNoteOn returns 0 = fail, 1 = success
delay(duration); // note length -see keys below for better solution
success = output.sendNoteOff(channel, note, velocity); // sendNoteOff returns 0 = fail, 1 = success
totalnotesplayed = totalnotesplayed + 1;
// Get the Integer Value of the Note
//println(note);

} catch (Exception e) {
}

try {

// Sets the Color Value of the Tip
tipR = int(map(note, 52, 88, 150, 255));
tipG = int(map(note, 52, 88, 30, 90));
tipB = int(map(note, 52, 88, 0, 10));

// Displays Red at the Highest points of the scale
if (note >= 80) {
tipR = 255;
tipG = 0;
tipB = 0;
// Note Yellow Value = tipR = 255, tipG = 255, tipB = 0;
}
// Moves the Globe along the X-Axiz
midi_mousex = int(map(note, 52, 88, 500, 2100));
midi_mousex = midi_mousex + log(random(50));
midi_mousey = int(map(note, 52, 88, 200,600));
midi_mousey = midi_mousey + log(random(50));

// Sets the Ampltude of the Waves
// High Frequency notes have greater impact on amplitude
//if (note < 65) {
int ampMap = int(map(note, 52, 88, 0, 50));
amplitude = ampMap/50 + .9;

//} else {
//int ampMap = int(map(note, 52, 88, 0, 50));
//amplitude = ampMap/50 + .9 + random(0, .65);

//}
// Controls the X and Y movements of the Main Bars
midi_mouse_glide_x = int(map(note, 52, 88, 300, 500));
midi_mouse_glide_y = int(map(note, 52, 88, 100, 300));
midi_mouse_glide_x = midi_mouse_glide_x + random(200, 300);
midi_mouse_glide_y = midi_mouse_glide_y + random(200, 300);
// Color of the Globe
// This creates a Red Value. Impacted by Note Value
colorval_r = 255;
colorval_g = 0;
colorval_b = int(map(note, 52, 70, 255, 0));
// The Color of the Bars. Mapped from Note Value
water_color = int(map(note, 52, 88, 0, 255));

// Control the Decay Rate of the Bars. These are Mapped to the Note Value
// The total notes played also affects the decay value
int decay = int(map(note, 52, 88, 3000, 6000));
decay = decay - totalnotesplayed;

// Control the Wave Velocity
float wave_velocity = int(map(note, 52, 88, 0.035, 2));
int trail_frequency = int(map(note, 52, 88, 30, 50));

} catch (Exception e) {
}

}
}

} catch (Exception e) {
}

} catch (Exception e) {
}
}

//***********
// END JL CODE
//***********

 

CONNECTING IT ALL UP:

 The wireless MIDI system should be placed in a robust but portable project case (see example below):

The instrument selected for this project was a Yamaha EZ-AG MIDI Guitar. The EZ-AG is no longer in production. However, any MIDI instrument is a potential candidate, assuming that the device looks can be battery powered.

Processing is not a particularly flexible tool for sound playback scenarios. Therefore, the MIDI information will likely need to be “piped” using a virtual MIDI configuration (e.g., via Audio MIDI Setup in OS/X) to GarageBand or a similar MIDI capable DAW. The process for wiring GargeBand specifically (though this could apply to other audio applications) is presented in detail at Hex705.  This element was essential to getting the “Pad” playback sounds required for the Lightness of Sound project.

 

BUILDING THE WIREMAP

To build the WireMap for the display, Mitzi followed this instructable:

http://www.instructables.com/id/How-to-Build-a-Wiremap/

We wanted to create smaller versions of it but the wiring for them proved to be a real challenge, the smaller the version, the more difficult is to put a wire without moving the one next to it.

 Mitzi Built 2 prototypes before settling with the a final version that is slightly smaller than the one described in the instructable.

TECHNICAL CHALLENGES:

 The biggest limitation of the approach identified is that of latency, as the SoftwareSerial library limits the data rate to 9600 bps. Players must be careful to avoid overloading the MIDI buffer.  The next iteration of this project would determine if a custom-built MIDI input module would improve MIDI transmission speeds and, therefore, reduce latency issues. Building a custom module would significantly reduce the cost of each transmission system and, therefore, enable more instruments to be employed.

 During testing the wireless MIDI circuit performed flawlessly for over 4 hours. However, on the day of the presentation the device became extremely unreliable (Murphy’s Law!). This may have been due to the number of XBee units in the performance space, as there were a large number of projects being exhibited at the time.  The lesson from this experience is that wireless communications can be very unpredictable. To compensate for this attention should be paid to isolating the XBee units from cross frequency traffic by modifying the XBee channel settings (there are 16 frequency bands) and by verifying that the devices have unique Pan ID’s. Finally, as each XBee has a per-programmed serial number it is possible to add this information to the Processing code to verify that the data packets are coming from the correct XBee units. Finally, it would have prudent to add Cyclical Redundancy Checks (CRC)/Error Checking to the communication routines. This would eliminate spurious note data.

In terms of the WireMap, building in in core foam allowed us to move the wires to calibrate it more easily that it would have been if it would have been constructed in acrylic, however, the fragility of it also made it easy to bow, causing some of the wires to miss the light of the proyector.

 PAST AND FUTURE OF THE LIGHTNESS OF SOUND:

We started the project with the idea of building 2 of more WireMaps that could communicate to each other via XBee’s  and  react to user interaction, however, the difficulties on building smaller versions of it in core foam, brought us to a better concept, the idea of building a bigger one that could react to music.

For the future we foresee that the ability to implement a wireless MIDI approach opens up virtually unlimited performance possibilities. In addition to utilizing existing MIDI technologies it is possible to design entirely new processes for eliciting audience interaction.  The first step in exploring the performance options would be to investigate the range of MIDI instruments currently available. All MIDI instruments are considered to be “controllers”. However, the devices tend to be used primarily as instruments or as dedicated input controllers (see examples below):

While the above MIDI controllers are physical devices, it is possible to represent their functionality in virtual form (i.e., in code).  This is significant as physical MIDI devices are often expensive and have strict power requirements (e.g., many MIDI devices are not portable).

 

The option of implementing MIDI controllers on mobile devices (tablets in particular) significantly lowers the cost of providing large numbers of devices to “performers”. In addition, “virtual” MIDI devices can be programmed with innovative control surface designs. There are a many virtual MIDI SDK’s available that enable custom control surface modeling. At present, Jazz Mutant Lemur is considered to be amongst the most flexible (see below), but designers are free to choose form a very large palette of audio/midi design components. Virtual MIDI devices typically communicate performance information via the Open Sound Protocol (OSC) or via dedicated MIDI interfaces. However, MIDI’s compactness enables it to be easily routed through any communication protocol.

Finally, it’s possible to develop entirely novel instruments and control surfaces for musical interaction. Custom instruments can be simple or complex, standalone or interconnected. Instruments can even be implemented as sculptural elements.

Source: http://log.liminastudio.com/itp/neurohedron

PERFORMANCE GOALS:

 

Challenge:

 Is it possible to create a collective musical/visual experience by providing the audience with MIDI instruments?

 

The participatory audio concept for the Lightness of Sound Project is a physical environment where the line between audience and performers is blurred.  It is envisioned as a space where primary performers (i.e., a “band” or DJ’s) encourage their “audience” to create a collective music experience.  This real-time musical generative environment would be visibly projected. See below:

It is hoped that this environment would encourage the audience to create complex and evolving musical performances. The projected content would act as a key element in the feedback loop as it would encourage the audience to push themselves musically.

 

One of the concepts for the project was to implement a class of MIDI instrument based on didgeridoos. As this physical instrument requires significant practice (breathing control skills etc.) to become proficient, the concept was adapted to enable a more approachable and portable solution (see below):

Source: Jeremy Littler. Concept for Electronic Didgeridoo

Additional instruments would be developed to support rhythmic control (portable drum pads and triggers) and melodic performance instruments. These items would be distributed to the “audience” in the performance space.  The performers would also be encouraged to download an application to their portable devices that would enable them to contribute to both the sound scape and projected elements via a virtual MIDI controller. Finally, sensors would be installed to pick-up the audience’s motion, temperature etc. This information would alter the soundscape and the visualization at the same time.

Comments are closed.