Assignment 8 Update: Video Interaction with Ultrasonic sensor

One of the things I wanted to explore in this class was the relationship between sensors and video playback, and what kinds of content would work in an installation with sensors. I did a simple experiment to update my week 8 assignment, which was to create an interaction between Arduino and Processing via a serial connection.

I used an Ultrasonic sensor to try out an interaction with a video that zooms in and out on a small note. Your proximity to the sensor makes the video scroll along its timeline, in relationship to sensor values. The video itself zooms in and out creating a relationship of distance with the viewer’s distance. I wanted to reverse the interaction, so that the closer you get to the screen and sensor, the further away the object/subject in the video gets. On a conceptual level, instead of facilitating your viewing experience, it problematizes it, in a way that may frustrate the viewer or depending on content, make it fun to try to read the message. This is the basic interaction:

The quick video I tested it with is here. What needs some adjustment is the sensor value code, but that would require testing in an empty space. It works well enough to express the interaction outlined.

Arduino Code
const int analogInPin = A0;  // Analog input pin that the potentiometer is attached to

int sensorValue = 0;        // value read from the sensor

void setup() {
Serial.begin(9600);
}

void loop() {
// read the analog in value:
sensorValue = analogRead(analogInPin);

// print the results to the serial monitor:
//Serial.print(“sensor = ” );
Serial.write(sensorValue);

delay(10);
}

Processing Code

// Creation and Computation Assignment Week 8
// Based on Exercise 16-5 from Learning Processing by Daniel Shiffman

import processing.video.*;
import processing.serial.*;

Serial myPort;        // The serial port
int videoControl;

import processing.video.*;

Movie myMovie;

void setup() {
size(320, 240);
myMovie = new Movie(this, “readmerev.mov”);
myMovie.loop();
println(Serial.list());
String portName = Serial.list()[0];  //initializing serial port
myPort = new Serial(this, portName, 9600);
}

void draw() {
float ratio = videoControl / (float) width;
myMovie.read();
image(myMovie, 0, 0);
}

void serialEvent(Serial myPort) {  // serial analog read
int inByte = myPort.read();
println(inByte);
videoControl=inByte;
float ratio = videoControl / (float) width;
myMovie.jump(ratio*myMovie.duration());
}

Protected: Final Project Proposal

This content is password protected. To view it please enter your password below:

Note Float! Cathy Chen & Maayan Cohen

OVERAL CONCEPT

A playful interactive musical experience for children We adopted the idea from Andre Michelle’s Tone Matrix http://lab.andre-michelle.com/tonematrix, and adapted it to a physical form. We both wanted to make something big, soft and physical for children to interact with and the Tone Matrix seemed to be a perfect fit for what we could start approaching.

APPROACH

Software and Hardware

At first we tried to find a library that will allow us to play notes on keyboard. Sound Cipher was the first one that we tried and was the one we ended up using.  http://explodingart.com/soundcipher/.

PROTOTYPES

Initial Software Prototype
The code basically imitated how the tone matrix works; however we wanted to make a smaller scale of the Tone Matrix as we considered the scope of the project is limited by the amount of time we are given; therefore we decided to make forty notes on keyboard initially.

8 notes on the column
5 notes on row (so five octaves)
with each note going incrementally on a pentatonic scale

so the pattern is:
first octave EGABDEGA
second octave: EGABDEGA
third octave: EGABDEGA
fourth octave: EGABDEGA
fifth  octave: EGABDEGA

we acquire the note number from this website: http://tonalsoft.com/pub/news/pitch-bend.aspx

After we got the keyboard working with the code, we went into hardware part of the operation. The main concept we used is “array of an array”

________
|__|__|__| <- an array with length 3 of floats = an “octave”
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
^
an array with length 8 of “octaves”

Here is the code for testing the various tone possibility for keypad:

/**
DIGF 6B01 Final Assignment Keyboard Test Version 4
November 11 2011
* Create a music keyboard from the qwerty keyboard.
* Capture key presses and map the ascii key number
* to a MIDI pitch number and play a note immediatly.
*
* A SoundCipher example by Andrew R. Brown
*/

import arb.soundcipher.*;

SoundCipher sc = new SoundCipher(this);

float [][] chord = new float [6][4];
boolean [][] pressed = new boolean [6][4];
int scaleCounter = 0;

void setup() {
  for (int i = 0; i < 6; i++) {
    for (int j = 0; j < 4; j++) {
      chord[i][j] = -1;
      pressed[i][j] = false;
    }
  }
}

void draw(){

    sc.playChord(chord[scaleCounter], 80, 0.3);

    scaleCounter++;

    if (scaleCounter == 6) {
        scaleCounter = 0;
    }

    delay(180);

}

void keyPressed()
{
  if (key == 'z') {
    if (!pressed[0][0]) {
      chord[0][0] = 64;
      pressed[0][0] = true;
    }
    else {
      chord[0][0] = -1;
      pressed[0][0] = false;
    }
  }
  else if (key == 'x') {
    if (!pressed[1][0]) {
      chord[1][0] = 67;
      pressed[1][0] = true;
    }
    else {
      chord[1][0] = -1;
      pressed[1][0] = false;
    }
  }
  else if (key == 'c') {
    if (!pressed[2][0]) {
      chord[2][0] = 69;
      pressed[2][0] = true;
      pressed[2][0] = true;
    }
    else {
      chord[2][0] = -1;
      pressed[2][0] = false;
    }
  }
  else if (key == 'v') {
    if (!pressed[3][0]) {
      chord[3][0] = 71;
      pressed[3][0] = true;
    }
    else {
      chord[3][0] = -1;
      pressed[3][0] = false;
    }
  }
  else if (key == 'b') {
    if (!pressed[4][0]) {
      chord[4][0] = 74;
      pressed[4][0] = true;
    }
    else {
      chord[4][0] = -1;
      pressed[4][0] = false;
    }
  }
  else if (key == 'n') {
    if (!pressed[5][0]) {
      chord[5][0] = 76;
      pressed[5][0] = true;
    }
    else {
      chord[5][0] = -1;
      pressed[5][0] = false;
    }
  }

   ////////////////////////////////////////

  else if (key == 'a') {
    if (!pressed[0][1]) {
      chord[0][1] = 79;
      pressed[0][1] = true;
    }
    else {
      chord[0][1] = -1;
      pressed[0][1] = false;
    }
  }
  else if (key == 's') {
    if (!pressed[1][1]) {
      chord[1][1] = 76;
      pressed[1][1] = true;
    }
    else {
      chord[1][1] = -1;
      pressed[1][1] = false;
    }
  }

  else if (key == 'd') {
    if (!pressed[2][1]) {
      chord[2][1] = 74;
      pressed[2][1] = true;
    }
    else {
      chord[2][1] = -1;
      pressed[2][1] = false;
    }
  }
  else if (key == 'f') {
    if (!pressed[3][1]) {
      chord[3][1] = 71;
      pressed[3][1] = true;
    }
    else {
      chord[3][1] = -1;
      pressed[3][1] = false;
    }
  }
  else if (key == 'g') {
    if (!pressed[4][1]) {
      chord[4][1] = 69;
      pressed[4][1] = true;
    }
    else {
      chord[4][1] = -1;
      pressed[4][1] = false;
    }
  }
  else if (key == 'h') {
    if (!pressed[5][1]) {
      chord[5][1] = 67;
      pressed[5][1] = true;
    }
    else {
      chord[5][1] = -1;
      pressed[5][1] = false;
    }
  }

   ////////////////////////////////////////

  else if (key == 'q') {
    if (!pressed[0][2]) {
      chord[0][2] = 69;
      pressed[0][2] = true;
    }
    else {
      chord[0][2] = -1;
      pressed[0][2] = false;
    }
  }
  else if (key == 'w') {
    if (!pressed[1][2]) {
      chord[1][2] = 71;
      pressed[1][2] = true;
    }
    else {
      chord[1][2] = -1;
      pressed[1][2] = false;
    }
  }
  else if (key == 'e') {
    if (!pressed[2][2]) {
      chord[2][2] = 74;
      pressed[2][2] = true;
    }
    else {
      chord[2][2] = -1;
      pressed[2][2] = false;
    }
  }
  else if (key == 'r') {
    if (!pressed[3][2]) {
      chord[3][2] = 76;
      pressed[3][2] = true;
    }
    else {
      chord[3][2] = -1;
      pressed[3][2] = false;
    }
  }
    else if (key == 't') {
    if (!pressed[4][2]) {
      chord[4][2] = 79;
      pressed[4][2] = true;
    }
    else {
      chord[4][2] = -1;
      pressed[4][2] = false;
    }
  }
    else if (key == 'y') {
    if (!pressed[5][2]) {
      chord[5][2] = 81;
      pressed[5][2] = true;
    }
    else {
      chord[5][2] = -1;
      pressed[5][2] = false;
    }
  }

   ////////////////////////////////////////

    else if (key == '1') {
    if (!pressed[0][3]) {
      chord[0][3] = 83;
      pressed[0][3] = true;
    }
    else {
      chord[0][3] = -1;
      pressed[0][3] = false;
    }
  }

    else if (key == '2') {
    if (!pressed[1][3]) {
      chord[1][3] = 81;
      pressed[1][3] = true;
    }
    else {
      chord[1][3] = -1;
      pressed[1][3] = false;
    }
  }
    else if (key == '3') {
    if (!pressed[2][3]) {
      chord[2][3] = 79;
      pressed[2][3] = true;
    }
    else {
      chord[2][3] = -1;
      pressed[2][3] = false;
    }
  }
    else if (key == '4') {
    if (!pressed[3][3]) {
      chord[3][3] = 76;
      pressed[3][3] = true;
    }
    else {
      chord[3][3] = -1;
      pressed[3][3] = false;
    }
  }
    else if (key == '5') {
    if (!pressed[4][3]) {
      chord[4][3] = 74;
      pressed[4][3] = true;
    }
    else {
      chord[4][3] = -1;
      pressed[4][3] = false;
    }
  }
    else if (key == '6') {
    if (!pressed[5][3]) {
      chord[5][3] = 71;
      pressed[5][3] = true;
    }
    else {
      chord[5][3] = -1;
      pressed[5][3] = false;
    }
  }
}

First Physical Interactive Choice (sensor): photoresistors

We tried connecting arduino with a sensor. Just for prototype, we used four photoresistors to activate the sound. However one problem we found with photoresistors is that it receives two values at the same time; thus if we asked it to turn on the sound if the sensor detects > 400, the photo resistor will detect 420, 422 at the same time, and it will very quickly turn on and turn off the sound. However we know that the sensor is reading the value and that it will turn on the music according to our code. So the first communication between the sensor and the code worked!

Second Physical Interactive Choice (sensor): Piezo Transducer

The second sensor we tried was piezo. We both felt piezo would be a perfect “sensor” for this project as it reacts with a knock or a touch. We grabbed a simple tutorial for piezo just to try to make it work: http://www.arduino.cc/en/Tutorial/KnockSensor Other examples that made us believe that piezo would work is: http://www.flickr.com/photos/peplop/3766646109/

However we encountered many problems with piezo transducers as it picks up noise even when we are not interacting with it. Eventually we asked Jim for help. Jim told us it is a mini microphone and that is why it’s picking up noises and signals even when we are not interacting with it. The piezo transducer then seems to be not working for us. Jim told us maybe a switch would do us well for our purpose. So we turned to our third choice of interaction

Third Physical Interactive Choice: Boolean Switch

Jim introduced us o micro switches. The main code that we tried is to use boolean to work with the switch. So when someone touches it once, the light/note will turn on and will keep turning on; however when someone touches it again, it will turn off.

     

Fourth Physical Interactive Choice: Reed Switch

Jim also introduced us to Reed Switches, which is a switch operated by an applied magnetic field; so there a magnet embedded in the switch. The reason why we turned to reed switches is because as we have to place the material on the conductive fabric in a certain manner in order for the interaction to work; however with the reed switch, it allows the interaction to work whichever way we put the individual block on the fabric.

Fifth Physical Interactive Choice: Simple Switch

However we discovered if we arrange the conductive fabric in a certain way so that both strips touch both sides of the conductive fabric on the block at the same time, the interaction would work no matter how we put the block down on the fabric. Therefore, due to the change of plan between a press interaction to a lift and drop interaction, we discarded the boolean code and changed to the basic switch code where when two conductive things touch, an interaction occurs. This became our final decision to build on our final prototype.

ps. we used a lot of aluminum foil for various prototypes. We like aluminum foils.

               

Final configurations for different shapes

          

MUSIC

We discovered that with the music going incrementally up the pentatonic scale in five, the music isn’t very interesting. So we devised few iterations of how the scale and octaves should perform, and plugged it in the keyboard codes:

The time signature that we ended up with is 2/6

B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81 B (xi) 83
A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81
G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76

 

B (xi) 83 A (la) 81 G (so) 79 E (mi) 76 D (re) 74 B (xi) 71
A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81
G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69 G (so) 67
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 

 

B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81 B (xi) 83
A (la) 81 G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69
G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69 G (so) 67
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76

 

PHYSICAL MAKING

We’ve considered several variations and arrangements for the project. We initially considered the grid that resembles Tone Matrix; however we wanted to make the project more mobile in some way so we thought of various possible interactions, such as placing individual notes separately on a tree the music will play according to the leaf pressed. In the end, we decided to have a board that has notes and we use several blocks to activate the notes. We thought of decorating the ‘board’ in an interesting narrative way so children can read stories while making music. The last idea we decided on is the round shape background and have various shapes of blocks for interaction.

Making of the Block

We’ve always wanted to use foam for shapes; therefore we tested round foams, square foams and decide why not use various shapes of foams to make the interaction more interesting and engaging. We also used various colours of felt fabric and lights, so that we consist of variety of shapes and colours. In total we devised eight shapes of each, six felt course, resulting in 24  notes in total. In each block, we wanted to make the block feel more firm and thus we combined two pieces of foam together to increase the volume of the block.

To decrease the number of digital pins on arduino mega, we decided to put the light with a battery holder and batter directly on the foams itself. On the night before the project is due, we found out that either the blocks are not flat enough or not heavy enough for the interaction between the blocks and the table to work desirably; we then decided to put rock inn between the two pieces of foams, and sew!

               

List of sewing for the blocks:
sewing the lights to the battery holder
sewing the fabric into various shapes
sewing the conductive fabric to the back of the shape
sewing the batter holder to the conductive fabric that is at the back of the shape

     

Making of the Round Fabric 

Trying to make a perfect circle out of a big piece of fabric was a challenge for us. Moreover, trying to measure forty eight strips equally was even more challenging on the not-so-perfect circle that we made.

     

Maayan ended up learning how to use a sewing machine and sewed all the individual strips to the round fabric whereas Cathy focused on other parts of the sewing that needed to be done by hand, as Cathy hated using the sewing machine since grade 8 Home Economic class when she sewed a big hole (unintentionally) on her final project, her boxer. Cathy is very thankful and proud of Maayan for dealing with that piece of complicated and sophisticated machine!

We connected the round fabric with conductive fabric and each end of conductive fabric has a clap sewed to it. The clap then is soldered to a wire, which is then connected to the board.

GENERAL ISSUES & CONCERNS

We wanted to hide the technology as much as possible; however this was difficult for us as we anticipated many wires hanging down from the round table. We then bought two portal boards as Jim taught us how to connect the portal boards together to the Arduino Mega.

     

Array Issue: With the arrangement of note, we can use array of an array, however as the number for each note (from the note library provided above) is different, and we have a different arrangement of scale, it became hard for us to devise an algorithm for the arrangement and configuration of the notes; therefore we hardcoded each note number in processing. This way it will be easier for us to change individual note.

Surprisingly, using an Arduino Mega was fairly difficult in some way. For some reason, not all the digital pins work properly when we try to plug everything connection in the pins. We had to try note by note and pin by pin to test which note works with which pin. Eventually we found out that  the even number pins work more cooperatively with us than the odd number pins. So we tested each even number pin one by one to ensure that the arrangement of notes and the interaction make sense.

     

FINAL ARDUINO CODE

/**
DIGF 6B01 Final Assignment Keyboard Test Version 17
November 29 2011 

*/

int numOctave = 4;  //number of rows we have
int numNotes = 6;  //number of columns we have
int totalSignal; //total number of switches
int activeButton = 0;

 int LEDoutput [] = {0, 0, 0, 0}; //LED lights in an array
 //int analogValue [] = {0, 0, 0, 0}; // switches in an array
 int button [] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; //the inital state of the swithces in an arary

void setup() {
    Serial.begin(9600); 

    totalSignal = numOctave * numNotes;

    Serial.println(totalSignal);

    for(int i = 0; i <  totalSignal; i++) {
      LEDoutput[i] = 2 + i;
      pinMode(LEDoutput[i], OUTPUT);
      pinMode(button[i], INPUT);

      //LEDoutput[i] = 9 - i*2; //the pin of the array, 9, 7, 5, 3

     // analogValue[i] = 0; //inital value of the switch
      //button[i] = true; // inital state of the switch
    }
}

void loop() {

  for(int i = 0; i < totalSignal; i++) {

    button[i] = 52 - i * 2;
    //map(analogRead(i), 0, 1023, 0, 255);

    if (button[i] == 20) {
      button[i] = 23;
    }

    /* when the switch value is bigger than one,
    if the button state is false, then make the button state true
    else, if the button state is true, make it false;*/

    activeButton = digitalRead(button[i]);
    Serial.write(activeButton);

    if(activeButton == HIGH){
      digitalWrite(LEDoutput[i], HIGH);
    } else {
      digitalWrite(LEDoutput[i], LOW);
    }
  }

 

FINAL PROCESSING CODE

/**
DIGF 6B01 Final Assignment Keyboard Test Version 13
November 29 2011 

Source: A SoundCipher Library by Andrew R. Brown
*/

import arb.soundcipher.*;  //soundcipher libraries
import processing.serial.*;

int numOctave = 4; // number of rows
int numNotes = 6; // number of columns
int totalSignal = numOctave * numNotes; // total number of notes

int currentByte = 0; // inByte counter
int [] inByte = new int [totalSignal];
/*
a new integer of array is called inByte;
it communicates between arduino and processing; informs on the number of signals
in this case, it is name of the individual notes
*/

int sensorValue; //

Serial myPort;
//boolean [][] on = new boolean [numNotes][numOctave]; //an array an of array
float [][] chord = new float [numNotes][numOctave]; //an array of an array
int scaleCounter = 0; //metronome 

SoundCipher sc = new SoundCipher (this); 

void setup(){
  String portName = Serial.list()[0];
  myPort = new Serial(this, portName, 9600);

  for(int i = 0; i < numNotes; i++){
   for(int j = 0; j < numOctave; j++){
     chord [i][j] = -1;
    // on [i][j] = false;
   }
 }

}

void draw(){

  /*
  sc is the new object. We access the function playChord from sc.
  chord is the entire canvas of notes
  scale counter acts as a metronome that accesses an individual column within the chord.
  chord [] gives you a column
  chord [] [] gives you specific note
  100 is the volume
  0.3 is the length of each note
  */
  sc.playChord(chord[scaleCounter], 100, 0.3);

  /*scaleCounter as the metronome keeps moving right until it reaches the total number
  fot notes (the end of the column), and it goes back to the beginning */
  scaleCounter++;

  if(scaleCounter == numNotes){
    scaleCounter = 0;
  }

  //determines the tempo
  delay(180);
}

void serialEvent(Serial myPort){

  /*
  the system tries to store every properties of the note in place by using currentByte
  this way the system ensures multiple notes can be played when the scale counter reaches
  the position of the notes. 

  */

  inByte[currentByte] = myPort.read();
  //println(inByte[currentByte]);
  currentByte++;

  /*thus only when the system contains the information of every notes (total signal), the system
  will perform the following tasks */

   if (currentByte == totalSignal) {

   if (inByte[0] > 0) { //if inByte [0] the analogValue is bigger than 0
        chord[0][0] = 64; //play the note 52;
      } else {
        chord[0][0] = -1; //don't play any note
      }

    if (inByte[1] > 0) {
        chord[1][0] = 67;
      } else {
        chord[1][0] = -1;
      }

    if (inByte[2] > 0) {
        chord[0][1] = 69;
      } else {
        chord[0][1] = -1;
      }

 if (inByte[3] > 0 ) {
        chord[1][1] = 71;
      } else {
        chord[1][1] = -1;
      }

 if (inByte[4] > 0) {
        chord[2][0] = 74;
      } else {
        chord[2][0] = -1;
      }

 if (inByte[5] > 0) {
        chord[2][1] = 76;
      } else {
        chord[2][1] = -1;
      }

     ///////////////////////////////

  if (inByte[6] > 0) {
        chord[3][0] = 79;
      } else {
        chord[3][0] = -1;
      }

  if (inByte[7] > 0) {
        chord[3][1] = 76;
      } else {
        chord[3][1] = -1;
      }

  if (inByte[8] > 0) {
        chord[4][0] = 74;
      } else {
        chord[4][0] = -1;
      }

  if (inByte[9] > 0) {
        chord[4][1] = 71;
      } else {
        chord[4][1] = -1;
      }
  if (inByte[10] > 0) {
        chord[5][0] = 69;
      } else {
        chord[5][0] = -1;
      }

 if (inByte[11] > 0) {
        chord[5][1] = 67;
      } else {
        chord[5][1] = -1;
      }

       ///////////////////////////////

 if (inByte[12] > 0) {
        chord[0][2] = 81;
      } else {
        chord[0][2] = -1;
      }      

 if (inByte[13] > 0) {
        chord[1][2] = 79;
      } else {
        chord[1][2] = -1;
      }

 if (inByte[14] > 0) {
        chord[2][2] = 76;
      } else {
        chord[2][2] = -1;
      }

  if (inByte[15] > 0) {
        chord[3][2] = 74;
      } else {
        chord[3][2] = -1;
      }

   if (inByte[16] > 0) {
        chord[4][2] = 71;
      } else {
        chord[4][2] = -1;
      }

 if (inByte[17] > 0) {
        chord[5][2] = 69;
      } else {
        chord[5][2] = -1;
      }
      ////////////////////////////////////////

     if (inByte[18] > 0) {
        chord[0][3] = 71;
      } else {
        chord[0][3] = -1;
      }

    if (inByte[19] > 0) {
        chord[1][3] = 74;
      } else {
        chord[1][3] = -1;
      }

    if (inByte[20] > 0) {
        chord[2][3] = 76;
      } else {
        chord[2][3] = -1;
      }

   if (inByte[21] > 0) {
        chord[3][3] = 79;
      } else {
        chord[3][3] = -1;
      }

   if (inByte[22] > 0) {
        chord[4][3] = 81;
      } else {
        chord[4][3] = -1;
      }

   if (inByte[23] > 0) {
        chord[5][3] = 83;
      } else {
        chord[5][3] = -1;
      }   

    println("1: " + inByte[0] + "    " + inByte[1]+ "    " + inByte[2]+ "    "+inByte[3]+ "    "+inByte[4]+ "    "+inByte[5]);
    println("2: " + inByte[6] + "    " + inByte[7]+ "    " + inByte[8]+ "    "+inByte[9]+ "    "+inByte[10]+ "    "+inByte[11]);
    println("3: " + inByte[12] + "    " + inByte[13]+ "    " + inByte[14]+ "    "+inByte[15]+ "    "+inByte[16]+ "    "+inByte[17]);
    println("4: " + inByte[18] + "    " + inByte[19]+ "    " + inByte[20]+ "    "+inByte[21]+ "    "+inByte[22]+ "    "+inByte[23]);

    currentByte = 0; //after performed all these, current goes to zero
  }

}

FUTURE IMPROVEMENT

We would like to incorporate lights in the project so that  the light could indicate the pace of the music.
We are also interested in incorporate letters with the experience to allow people to deliver interesting messages while interacting in making the music based on pentatonic scale.

Protected: USB Keypad

This content is password protected. To view it please enter your password below:

final_felt petals

just a quick FYI … I started making flower petals and sowing muscle wire into them.

 

final_1st felt experiment

 

 

a quick muscle wire experiment

IMG_0150

Generative Harp… and Robo Music Box.

 

 


 

I set out to create a sensor based version of my WEEK 3 experiment using light sensors. The objective was to produce a generative music instrument, like a wind chime, that would play a 9 note melody I had written. Ideally, such an instrument would be passive, set up as an ambient experience in a hall or hallway type environment. For the purposes of this demo, the trigger sensitivity is tuned to user gestures (swipes changing the light readings), as opposed to subtle light changes (which would be more analogous to a wind chime… and totally possible).

In order to accomplish this goal, I leaned heavily on my WEEK 7 light sensor experiments (though the sensor I chose was not part of that test) and my WEEK 8 Processing + Arduino creation.

STEP 1 – REWRITE THE ELASTIC STRINGS CODE

This took much longer than I had anticipated; creating an object out of my original sketch was a challenge as I had to convert the many (and I mean MANY) arrays that stored all of the individual elastic variables in to a single class that would represent each elastic. As I mentioned I leaned heavily on my WEEK 8 creation as it was my first object oriented sketch. I knew I wanted to make this Generative Harp prior to WEEK 8, so I purposefully used the Hoops creation to explore OOP and Classes, but I also used it to figure out how I needed to structure my code in order to trigger/modify a single object while the other objects execute their properties uninterrupted. After numerous pots of tea, I managed to convert WEEK 3 into a SKETCH reminiscent of WEEK 8, except with the elastics still trigger by the mouse. Sensor input was the next step.

STEP 2 – CHOOSE A LIGHT SENSOR

I purchased 6 TSL235R Light to Frequency sensors mainly because were more sensitive than a photoresistor, and cheaper than a TEMT6000 (though I bought a few more of those as back-ups along with a couple proximity sensors). After testing the TSL235Rs I quickly realized that I had luckily made the perfect choice as these sensors were particularly sensitive to low light. Where the TEMT6000 reads light levels as an analog output with a positive correlation (more light = higher value), the TSL235R sensor is digital and outputs a negative correlation to the light reading; an average daily light lit indoor room read beween 20 and 50 with pure daylight returning a value of 1. This was all very fortunate because I imagined these sensors as light based triggers based on a reduction in light. The TSL235R produced a wide range of low light values by covering it with my hand, anywhere between 200 to 1500+! This was nearly identical to the range of the TEMT6000, only inverted. Perfect.

I now had to program the Arduino to average the 6 sensors upon booting in order to calibrate system based on the environment. I modified the Smoothing example code and created boot averages for each sensor and for all 6 sensors. I then had to send these 14 variables (6 live sensors, 6 sensor boot averages, 1 all sensor boot average, and 1 live all sensor average) to Processing as input for my newly coded elastics.

STEP 3 – CREATE LIGHT TRIGGERS

I had learned early on in the semester that coding hard values into your code was ultimately very limiting because it wasn’t scalable. Consequently, my code is riddled with variables in order to allow for scalability, but also to account for the calibration I had just programmed above in my Arduino sketch. Without going into great detail, the individual sensor averages are used as the control variables by which any change to the live sensor is judged. This is how I created light sensor triggers to replace the mouse interactions found in the object oriented sketch in STEP 1. If a swipe was detected, the FreqCheck class would move the Elastic array forward one spot, send the sensor data through to the Elastic class instance corresponding to that array location in order to draw and animate the bezier. Simply put, this substitutes the mouse for light sensors, triggering X AXIS movement.

Because the light TSL235R sensors were so sensitive to low light, I was able to use the live all sensor average variable as a Y AXIS and Colour (R: Bezier X point, G: Bezier Y Point, B: Light Average) manipulator. I built a function that checks for changes in the all sensor average base on the all sensor boot average control, exactly the same method as above.

STEP 4 – TRIGGER MIDI NOTES

There are 3 different libraries available for MIDI in Processing, but I chose to use the MidiBUS for no reason other than their example worked and the code was legible. I created an array of notes, fed it through their demo code, and soon had my melody playing on my virtual instrument (IK Multimedia’s Sonik Synth running a wonderful Pianet sample, an electric keyboard I used to own and one I still very much love the sound of). The code included a delay between the onNote and offNote commands, a delay which would also pause my animation were I to copy it directly in to my sketch. But unfortunately, it’s not as simple as just removing the delay command as a onNote paired with an offNote on the next line triggers a MIDI note so short that barely any sound is produced. I thus had to remove the offNote command. This may seem obvious, but its actually a big problem if it is not accounted for as without an offNote command a sound with an infinite sustain (like an organ) would play forever. I thus had to make sure I chose a tine like sound with a strong attack and a relatively fast decay.

I ran into some trouble when trying to make the melody play in stereo. I knew I had to split the L and R notes in to separate MIDI channels, but for some reason inside Logic and Ableton Live, the virtual instrument would only read channel 1 (or channel 0 in Processing). I thus had to go into Audio MIDI Setup and create a second IAC Driver, double up my MidiBus instances in my code (one for each channel) and duplicate my virtual instruments on Ableton. Once I had figured this out (I’m still mad at Logic for not handling channel command properly!) I was able to trigger a hard panned left version of the Pianet, and a hard panned right version of the Pianet. Now I could have used a sound generation library in Processing, but I wanted to craft the sonic experience with a little more care, so for me this extra frustration was worth it: the Pianet is triggered in stereo while being run through a nice reverb and a channel compressor and a hard master limiter.

It was a slightly more complicated than a simple copy and paste job when inserting the above MIDI code into my main processing sketch. I knew I wanted the MIDI notes to be triggered at maximum elastic extension, so finding the right place to insert the command was simple (the X AXIS direction switch function), but I spent quite a bit of time ensuring that it only triggered once (my directionX “if” constraints were not tight enough to ensure that the direction switch only happened once along the animation, producing multiple consecutive trues or falses before reaching the maximum elastic extent, and truly flipping). Once I refined the code, the whole system sang… almost exactly as I imagined it would.

PRESENTATION

I struggled trying to find a way of presenting the system that was clean and functional. The sensors are so small, and needed to be spread out in a such a way that I became very limited in my mounting options. I decided to embrace the wires because my animation is all about moving wires. I ended up buying a pack of suction cups which held the braided wire quite nicely, and most importantly adhered to the HDTV perfectly. In the end I liked the look, but I also liked the idea that the sensors could be moved. I think if I were to refine this idea a bit more I would design it so that the sensors were not set up as a left to right continuum, but rather as independent triggers that anyone can move anywhere they want. Doing so would also bring the system that much closer to the intended generative purpose by removing the direct cause and effect of triggering sensors direct affixed to the displaying screen.

 

 

 

PROCESSING CODE

TAB 1: Main ProgramTAB 2: Elastic ClassTAB 3: Frequency Check Class

ARDUINO CODE

Light Sensor Read + Average

------------------------------------------------------------

ROBO MUSIC BOX

 

http://vimeo.com/33010630




 

My ultimate goal for this whole project was to create a system that would generate music. Although the Generative Harp is capable (it was playing on its own in bright afternoon sun light when pointed out the window), I wanted to explore a more physical system that would also respond to light changes, specifically ambient light. I bought a DIY music box and scored the full melody on one of the supplied sheets. I then hooked a continuous 360 degree servo up to the crank (crudely joined by hot glue), plugged in a TEMT6000 to an Arduino analog pin and sat back. The code in the Arduino is constructed to store the difference between 2 averaged light readings in to a container variable that when filled, would trigger two turns of the servo, and consequently a note or two from the music box. The TEMT6000 is fantastic at measuring ambient light in a room (don’t point it out a window as it will max out easily on direct sunlight). The problem came from the competing noise of the motor: you couldn’t hear the music box! This much I anticipated. Yet this rather crude exploration only further reinforced the beauty of the wind chime by reminding me that it isn’t only about translating one type of wave (wind) in to another (sound), in my case light (sensor) to sound (music box), but also about doing so without drawing attention to how. Though endearing in its own way, Robo Music Box wasn’t quite as magical an experience as I had hoped… I know this because when the Generative Harp played on its own without anyone around it did feel and sound like there was magic in the room.

 

ARDUINO CODE

 Ambient Light Read + Average + Store Difference + Trigger Motor

------------------------------------------------------------


THE MELODY

 

http://vimeo.com/33010378

Final Post!

I’m pleased with how the electronic etch-a-sketch worked out. It is amazing how complicated even the simplest projects can be – there were lots of things to trouble shoot along the way. One thing helped a lot to simplify my code. Jim suggested that I import the standard firmata library and use processing to read directly from the arduino board instead of making a serial connection (if I understand that right?). It worked well and I have four sketches to share with glitches in every one. I started by drawing the etch-a-sketch in Illustrator, saving it in Photoshop as a jpeg file and used that as a background image in my Processing sketch. Then I needed to constrain the x and y co-ordinates to stay within the visual parameters of the etch-sketch frame. The issue with the first sketch (most like the original toy) is that in order to get a smooth line I need to write the code for “line” after the code that reads the analogRead code and the constrain code. I cannot figure out why the line begins at (0, 0). During the rest of the draw loop it stays within the constrain function. If I want the line to begin where the x and y co-ordinates of the pentiometers, then the code for “line” must come BEFORE the analogRead code. However, the line is choppy and reacts to the speed of the pentiometer readings.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Then I had some fun with colour – tried this with the thin line and a thicker more transparent line.

 

 

 

 

 

 

 

 

 

 

 

 

 

To finish off the concept, I wanted to erase the drawing by turning the etch-a-sketch upside down. I wired in a tilt ball sensor and attached this to digital pin 2 and then altered a piece of code I found that would read whether this “switch” was either HIGH or LOW. If HIGH, then the screen would fill with a light grey rectangle. Else (when turned upside down) the rectangle would fill would change to black. It works except that this if/else statement is written within a “for” loop. So the program only runs the draw loop once and then just keeps drawing the background rectangle. I  cannot tell you how many different ways I tried to “fix” this. This includes trying to run this part of the program in “void setup” so the “void draw” loop runs on top. I wrote drawGreyRectangle and drawBlackRecctangle as functions to call within the Void draw loop and tried to write code that did not use a the “for” statement to activate the switch but this would not work with the Arduino. So after investing 10 hours with not solution I am going to ask for help…. maybe someone tonight can solve the mystery! The code is below…

 

 

final_ PWM/Nitinol circuit – success!

*big thanks to Jim Ruxton for help with the PWM/Nitinol circuit.

**PS Jim, I could not find an IRL530 – but the IRF530 seems to work fine.

—————————————

Step two was hooking a piece of Memory Wire up to an Arduino.

A very helpful read was the MIT PhD thesis of Marcelo Coelho, who worked on the Kukkia Flower back in Montreal. Marcelo’s work was specifically helpful in two ways.

1) techniques for memory shaping Nitinol

2) circuitry for working with Nitinol.

The take-away was that I should consider an independent 12V power supply for the Nitinol and I needed something called a MOSFET N-ch (a transistor).

Wikipedia described the MOSFET as having “a voltage on the oxide-insulated gate electrode can induce a conducting channel between the two other contacts called source and drain.

I used the PWM code from the Analogue lab to run current through the muscle wire, starting at 0  and going up to 1023.

Here’s a diagram of the PWM/Muscle wire circuit.

Now it’s time to design petal power (anchor Nitinol into felt and see what works).

not quite sure what I am going to do…

Testing Testing…

Setting up the potentiometers to control the horizontal and vertical lines is fairly basic and should be relatively simple but will take you thru my odyssey nevertheless.

Test 1: I used the Arduino code from the Graphing a Sensor lab and proceeded to over-engineer it. In the Dimmer lab, I was unhappy that the potentiometer did not move the ball the full width of the screen and I thought I could overcome this by mapping the analogRead to the width of my screen dimension (1200 pixels). I also purchased what I thought was a better potentiometer than the one in our kits, in case that was the issue. Below is the code and the results. It worked in that I could produce a horizontal line. The problem however is I get a dotted line if I rotate the potentiometer to quickly.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Test 2: I tried re-purposing the code from the Graphing a Sensor lab and it worked much better. It now draws a smooth line no matter how fast the rotation. But of course there is a new problem to solve. When I open the sketch, the line draws from right to left, yet the potentiometer rotation is from left to right. The code is below – I’m sure the solution is blindingly obvious but I’m just tired trying right now. Time for a break…

And after staring at it again, I see the error in the Processing code… the fix is highlighted below in yellow. Works now – left to right rotation = line drawn from left to right on screen.