Note Float! Cathy Chen & Maayan Cohen

OVERAL CONCEPT

A playful interactive musical experience for children We adopted the idea from Andre Michelle’s Tone Matrix http://lab.andre-michelle.com/tonematrix, and adapted it to a physical form. We both wanted to make something big, soft and physical for children to interact with and the Tone Matrix seemed to be a perfect fit for what we could start approaching.

APPROACH

Software and Hardware

At first we tried to find a library that will allow us to play notes on keyboard. Sound Cipher was the first one that we tried and was the one we ended up using.  http://explodingart.com/soundcipher/.

PROTOTYPES

Initial Software Prototype
The code basically imitated how the tone matrix works; however we wanted to make a smaller scale of the Tone Matrix as we considered the scope of the project is limited by the amount of time we are given; therefore we decided to make forty notes on keyboard initially.

8 notes on the column
5 notes on row (so five octaves)
with each note going incrementally on a pentatonic scale

so the pattern is:
first octave EGABDEGA
second octave: EGABDEGA
third octave: EGABDEGA
fourth octave: EGABDEGA
fifth  octave: EGABDEGA

we acquire the note number from this website: http://tonalsoft.com/pub/news/pitch-bend.aspx

After we got the keyboard working with the code, we went into hardware part of the operation. The main concept we used is “array of an array”

________
|__|__|__| <- an array with length 3 of floats = an “octave”
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
|__|__|__|
^
an array with length 8 of “octaves”

Here is the code for testing the various tone possibility for keypad:

/**
DIGF 6B01 Final Assignment Keyboard Test Version 4
November 11 2011
* Create a music keyboard from the qwerty keyboard.
* Capture key presses and map the ascii key number
* to a MIDI pitch number and play a note immediatly.
*
* A SoundCipher example by Andrew R. Brown
*/

import arb.soundcipher.*;

SoundCipher sc = new SoundCipher(this);

float [][] chord = new float [6][4];
boolean [][] pressed = new boolean [6][4];
int scaleCounter = 0;

void setup() {
  for (int i = 0; i < 6; i++) {
    for (int j = 0; j < 4; j++) {
      chord[i][j] = -1;
      pressed[i][j] = false;
    }
  }
}

void draw(){

    sc.playChord(chord[scaleCounter], 80, 0.3);

    scaleCounter++;

    if (scaleCounter == 6) {
        scaleCounter = 0;
    }

    delay(180);

}

void keyPressed()
{
  if (key == 'z') {
    if (!pressed[0][0]) {
      chord[0][0] = 64;
      pressed[0][0] = true;
    }
    else {
      chord[0][0] = -1;
      pressed[0][0] = false;
    }
  }
  else if (key == 'x') {
    if (!pressed[1][0]) {
      chord[1][0] = 67;
      pressed[1][0] = true;
    }
    else {
      chord[1][0] = -1;
      pressed[1][0] = false;
    }
  }
  else if (key == 'c') {
    if (!pressed[2][0]) {
      chord[2][0] = 69;
      pressed[2][0] = true;
      pressed[2][0] = true;
    }
    else {
      chord[2][0] = -1;
      pressed[2][0] = false;
    }
  }
  else if (key == 'v') {
    if (!pressed[3][0]) {
      chord[3][0] = 71;
      pressed[3][0] = true;
    }
    else {
      chord[3][0] = -1;
      pressed[3][0] = false;
    }
  }
  else if (key == 'b') {
    if (!pressed[4][0]) {
      chord[4][0] = 74;
      pressed[4][0] = true;
    }
    else {
      chord[4][0] = -1;
      pressed[4][0] = false;
    }
  }
  else if (key == 'n') {
    if (!pressed[5][0]) {
      chord[5][0] = 76;
      pressed[5][0] = true;
    }
    else {
      chord[5][0] = -1;
      pressed[5][0] = false;
    }
  }

   ////////////////////////////////////////

  else if (key == 'a') {
    if (!pressed[0][1]) {
      chord[0][1] = 79;
      pressed[0][1] = true;
    }
    else {
      chord[0][1] = -1;
      pressed[0][1] = false;
    }
  }
  else if (key == 's') {
    if (!pressed[1][1]) {
      chord[1][1] = 76;
      pressed[1][1] = true;
    }
    else {
      chord[1][1] = -1;
      pressed[1][1] = false;
    }
  }

  else if (key == 'd') {
    if (!pressed[2][1]) {
      chord[2][1] = 74;
      pressed[2][1] = true;
    }
    else {
      chord[2][1] = -1;
      pressed[2][1] = false;
    }
  }
  else if (key == 'f') {
    if (!pressed[3][1]) {
      chord[3][1] = 71;
      pressed[3][1] = true;
    }
    else {
      chord[3][1] = -1;
      pressed[3][1] = false;
    }
  }
  else if (key == 'g') {
    if (!pressed[4][1]) {
      chord[4][1] = 69;
      pressed[4][1] = true;
    }
    else {
      chord[4][1] = -1;
      pressed[4][1] = false;
    }
  }
  else if (key == 'h') {
    if (!pressed[5][1]) {
      chord[5][1] = 67;
      pressed[5][1] = true;
    }
    else {
      chord[5][1] = -1;
      pressed[5][1] = false;
    }
  }

   ////////////////////////////////////////

  else if (key == 'q') {
    if (!pressed[0][2]) {
      chord[0][2] = 69;
      pressed[0][2] = true;
    }
    else {
      chord[0][2] = -1;
      pressed[0][2] = false;
    }
  }
  else if (key == 'w') {
    if (!pressed[1][2]) {
      chord[1][2] = 71;
      pressed[1][2] = true;
    }
    else {
      chord[1][2] = -1;
      pressed[1][2] = false;
    }
  }
  else if (key == 'e') {
    if (!pressed[2][2]) {
      chord[2][2] = 74;
      pressed[2][2] = true;
    }
    else {
      chord[2][2] = -1;
      pressed[2][2] = false;
    }
  }
  else if (key == 'r') {
    if (!pressed[3][2]) {
      chord[3][2] = 76;
      pressed[3][2] = true;
    }
    else {
      chord[3][2] = -1;
      pressed[3][2] = false;
    }
  }
    else if (key == 't') {
    if (!pressed[4][2]) {
      chord[4][2] = 79;
      pressed[4][2] = true;
    }
    else {
      chord[4][2] = -1;
      pressed[4][2] = false;
    }
  }
    else if (key == 'y') {
    if (!pressed[5][2]) {
      chord[5][2] = 81;
      pressed[5][2] = true;
    }
    else {
      chord[5][2] = -1;
      pressed[5][2] = false;
    }
  }

   ////////////////////////////////////////

    else if (key == '1') {
    if (!pressed[0][3]) {
      chord[0][3] = 83;
      pressed[0][3] = true;
    }
    else {
      chord[0][3] = -1;
      pressed[0][3] = false;
    }
  }

    else if (key == '2') {
    if (!pressed[1][3]) {
      chord[1][3] = 81;
      pressed[1][3] = true;
    }
    else {
      chord[1][3] = -1;
      pressed[1][3] = false;
    }
  }
    else if (key == '3') {
    if (!pressed[2][3]) {
      chord[2][3] = 79;
      pressed[2][3] = true;
    }
    else {
      chord[2][3] = -1;
      pressed[2][3] = false;
    }
  }
    else if (key == '4') {
    if (!pressed[3][3]) {
      chord[3][3] = 76;
      pressed[3][3] = true;
    }
    else {
      chord[3][3] = -1;
      pressed[3][3] = false;
    }
  }
    else if (key == '5') {
    if (!pressed[4][3]) {
      chord[4][3] = 74;
      pressed[4][3] = true;
    }
    else {
      chord[4][3] = -1;
      pressed[4][3] = false;
    }
  }
    else if (key == '6') {
    if (!pressed[5][3]) {
      chord[5][3] = 71;
      pressed[5][3] = true;
    }
    else {
      chord[5][3] = -1;
      pressed[5][3] = false;
    }
  }
}

First Physical Interactive Choice (sensor): photoresistors

We tried connecting arduino with a sensor. Just for prototype, we used four photoresistors to activate the sound. However one problem we found with photoresistors is that it receives two values at the same time; thus if we asked it to turn on the sound if the sensor detects > 400, the photo resistor will detect 420, 422 at the same time, and it will very quickly turn on and turn off the sound. However we know that the sensor is reading the value and that it will turn on the music according to our code. So the first communication between the sensor and the code worked!

Second Physical Interactive Choice (sensor): Piezo Transducer

The second sensor we tried was piezo. We both felt piezo would be a perfect “sensor” for this project as it reacts with a knock or a touch. We grabbed a simple tutorial for piezo just to try to make it work: http://www.arduino.cc/en/Tutorial/KnockSensor Other examples that made us believe that piezo would work is: http://www.flickr.com/photos/peplop/3766646109/

However we encountered many problems with piezo transducers as it picks up noise even when we are not interacting with it. Eventually we asked Jim for help. Jim told us it is a mini microphone and that is why it’s picking up noises and signals even when we are not interacting with it. The piezo transducer then seems to be not working for us. Jim told us maybe a switch would do us well for our purpose. So we turned to our third choice of interaction

Third Physical Interactive Choice: Boolean Switch

Jim introduced us o micro switches. The main code that we tried is to use boolean to work with the switch. So when someone touches it once, the light/note will turn on and will keep turning on; however when someone touches it again, it will turn off.

     

Fourth Physical Interactive Choice: Reed Switch

Jim also introduced us to Reed Switches, which is a switch operated by an applied magnetic field; so there a magnet embedded in the switch. The reason why we turned to reed switches is because as we have to place the material on the conductive fabric in a certain manner in order for the interaction to work; however with the reed switch, it allows the interaction to work whichever way we put the individual block on the fabric.

Fifth Physical Interactive Choice: Simple Switch

However we discovered if we arrange the conductive fabric in a certain way so that both strips touch both sides of the conductive fabric on the block at the same time, the interaction would work no matter how we put the block down on the fabric. Therefore, due to the change of plan between a press interaction to a lift and drop interaction, we discarded the boolean code and changed to the basic switch code where when two conductive things touch, an interaction occurs. This became our final decision to build on our final prototype.

ps. we used a lot of aluminum foil for various prototypes. We like aluminum foils.

               

Final configurations for different shapes

          

MUSIC

We discovered that with the music going incrementally up the pentatonic scale in five, the music isn’t very interesting. So we devised few iterations of how the scale and octaves should perform, and plugged it in the keyboard codes:

The time signature that we ended up with is 2/6

B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81 B (xi) 83
A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81
G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76

 

B (xi) 83 A (la) 81 G (so) 79 E (mi) 76 D (re) 74 B (xi) 71
A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81
G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69 G (so) 67
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76 

 

B (xi) 71 D (re) 74 E (mi) 76 G (so) 79 A (la) 81 B (xi) 83
A (la) 81 G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69
G (so) 79 E (mi) 76 D (re) 74 B (xi) 71 A (la) 69 G (so) 67
E (mi) 64 G (so) 67 A (la) 69 B (xi) 71 D (re) 74 E (mi) 76

 

PHYSICAL MAKING

We’ve considered several variations and arrangements for the project. We initially considered the grid that resembles Tone Matrix; however we wanted to make the project more mobile in some way so we thought of various possible interactions, such as placing individual notes separately on a tree the music will play according to the leaf pressed. In the end, we decided to have a board that has notes and we use several blocks to activate the notes. We thought of decorating the ‘board’ in an interesting narrative way so children can read stories while making music. The last idea we decided on is the round shape background and have various shapes of blocks for interaction.

Making of the Block

We’ve always wanted to use foam for shapes; therefore we tested round foams, square foams and decide why not use various shapes of foams to make the interaction more interesting and engaging. We also used various colours of felt fabric and lights, so that we consist of variety of shapes and colours. In total we devised eight shapes of each, six felt course, resulting in 24  notes in total. In each block, we wanted to make the block feel more firm and thus we combined two pieces of foam together to increase the volume of the block.

To decrease the number of digital pins on arduino mega, we decided to put the light with a battery holder and batter directly on the foams itself. On the night before the project is due, we found out that either the blocks are not flat enough or not heavy enough for the interaction between the blocks and the table to work desirably; we then decided to put rock inn between the two pieces of foams, and sew!

               

List of sewing for the blocks:
sewing the lights to the battery holder
sewing the fabric into various shapes
sewing the conductive fabric to the back of the shape
sewing the batter holder to the conductive fabric that is at the back of the shape

     

Making of the Round Fabric 

Trying to make a perfect circle out of a big piece of fabric was a challenge for us. Moreover, trying to measure forty eight strips equally was even more challenging on the not-so-perfect circle that we made.

     

Maayan ended up learning how to use a sewing machine and sewed all the individual strips to the round fabric whereas Cathy focused on other parts of the sewing that needed to be done by hand, as Cathy hated using the sewing machine since grade 8 Home Economic class when she sewed a big hole (unintentionally) on her final project, her boxer. Cathy is very thankful and proud of Maayan for dealing with that piece of complicated and sophisticated machine!

We connected the round fabric with conductive fabric and each end of conductive fabric has a clap sewed to it. The clap then is soldered to a wire, which is then connected to the board.

GENERAL ISSUES & CONCERNS

We wanted to hide the technology as much as possible; however this was difficult for us as we anticipated many wires hanging down from the round table. We then bought two portal boards as Jim taught us how to connect the portal boards together to the Arduino Mega.

     

Array Issue: With the arrangement of note, we can use array of an array, however as the number for each note (from the note library provided above) is different, and we have a different arrangement of scale, it became hard for us to devise an algorithm for the arrangement and configuration of the notes; therefore we hardcoded each note number in processing. This way it will be easier for us to change individual note.

Surprisingly, using an Arduino Mega was fairly difficult in some way. For some reason, not all the digital pins work properly when we try to plug everything connection in the pins. We had to try note by note and pin by pin to test which note works with which pin. Eventually we found out that  the even number pins work more cooperatively with us than the odd number pins. So we tested each even number pin one by one to ensure that the arrangement of notes and the interaction make sense.

     

FINAL ARDUINO CODE

/**
DIGF 6B01 Final Assignment Keyboard Test Version 17
November 29 2011 

*/

int numOctave = 4;  //number of rows we have
int numNotes = 6;  //number of columns we have
int totalSignal; //total number of switches
int activeButton = 0;

 int LEDoutput [] = {0, 0, 0, 0}; //LED lights in an array
 //int analogValue [] = {0, 0, 0, 0}; // switches in an array
 int button [] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; //the inital state of the swithces in an arary

void setup() {
    Serial.begin(9600); 

    totalSignal = numOctave * numNotes;

    Serial.println(totalSignal);

    for(int i = 0; i <  totalSignal; i++) {
      LEDoutput[i] = 2 + i;
      pinMode(LEDoutput[i], OUTPUT);
      pinMode(button[i], INPUT);

      //LEDoutput[i] = 9 - i*2; //the pin of the array, 9, 7, 5, 3

     // analogValue[i] = 0; //inital value of the switch
      //button[i] = true; // inital state of the switch
    }
}

void loop() {

  for(int i = 0; i < totalSignal; i++) {

    button[i] = 52 - i * 2;
    //map(analogRead(i), 0, 1023, 0, 255);

    if (button[i] == 20) {
      button[i] = 23;
    }

    /* when the switch value is bigger than one,
    if the button state is false, then make the button state true
    else, if the button state is true, make it false;*/

    activeButton = digitalRead(button[i]);
    Serial.write(activeButton);

    if(activeButton == HIGH){
      digitalWrite(LEDoutput[i], HIGH);
    } else {
      digitalWrite(LEDoutput[i], LOW);
    }
  }

 

FINAL PROCESSING CODE

/**
DIGF 6B01 Final Assignment Keyboard Test Version 13
November 29 2011 

Source: A SoundCipher Library by Andrew R. Brown
*/

import arb.soundcipher.*;  //soundcipher libraries
import processing.serial.*;

int numOctave = 4; // number of rows
int numNotes = 6; // number of columns
int totalSignal = numOctave * numNotes; // total number of notes

int currentByte = 0; // inByte counter
int [] inByte = new int [totalSignal];
/*
a new integer of array is called inByte;
it communicates between arduino and processing; informs on the number of signals
in this case, it is name of the individual notes
*/

int sensorValue; //

Serial myPort;
//boolean [][] on = new boolean [numNotes][numOctave]; //an array an of array
float [][] chord = new float [numNotes][numOctave]; //an array of an array
int scaleCounter = 0; //metronome 

SoundCipher sc = new SoundCipher (this); 

void setup(){
  String portName = Serial.list()[0];
  myPort = new Serial(this, portName, 9600);

  for(int i = 0; i < numNotes; i++){
   for(int j = 0; j < numOctave; j++){
     chord [i][j] = -1;
    // on [i][j] = false;
   }
 }

}

void draw(){

  /*
  sc is the new object. We access the function playChord from sc.
  chord is the entire canvas of notes
  scale counter acts as a metronome that accesses an individual column within the chord.
  chord [] gives you a column
  chord [] [] gives you specific note
  100 is the volume
  0.3 is the length of each note
  */
  sc.playChord(chord[scaleCounter], 100, 0.3);

  /*scaleCounter as the metronome keeps moving right until it reaches the total number
  fot notes (the end of the column), and it goes back to the beginning */
  scaleCounter++;

  if(scaleCounter == numNotes){
    scaleCounter = 0;
  }

  //determines the tempo
  delay(180);
}

void serialEvent(Serial myPort){

  /*
  the system tries to store every properties of the note in place by using currentByte
  this way the system ensures multiple notes can be played when the scale counter reaches
  the position of the notes. 

  */

  inByte[currentByte] = myPort.read();
  //println(inByte[currentByte]);
  currentByte++;

  /*thus only when the system contains the information of every notes (total signal), the system
  will perform the following tasks */

   if (currentByte == totalSignal) {

   if (inByte[0] > 0) { //if inByte [0] the analogValue is bigger than 0
        chord[0][0] = 64; //play the note 52;
      } else {
        chord[0][0] = -1; //don't play any note
      }

    if (inByte[1] > 0) {
        chord[1][0] = 67;
      } else {
        chord[1][0] = -1;
      }

    if (inByte[2] > 0) {
        chord[0][1] = 69;
      } else {
        chord[0][1] = -1;
      }

 if (inByte[3] > 0 ) {
        chord[1][1] = 71;
      } else {
        chord[1][1] = -1;
      }

 if (inByte[4] > 0) {
        chord[2][0] = 74;
      } else {
        chord[2][0] = -1;
      }

 if (inByte[5] > 0) {
        chord[2][1] = 76;
      } else {
        chord[2][1] = -1;
      }

     ///////////////////////////////

  if (inByte[6] > 0) {
        chord[3][0] = 79;
      } else {
        chord[3][0] = -1;
      }

  if (inByte[7] > 0) {
        chord[3][1] = 76;
      } else {
        chord[3][1] = -1;
      }

  if (inByte[8] > 0) {
        chord[4][0] = 74;
      } else {
        chord[4][0] = -1;
      }

  if (inByte[9] > 0) {
        chord[4][1] = 71;
      } else {
        chord[4][1] = -1;
      }
  if (inByte[10] > 0) {
        chord[5][0] = 69;
      } else {
        chord[5][0] = -1;
      }

 if (inByte[11] > 0) {
        chord[5][1] = 67;
      } else {
        chord[5][1] = -1;
      }

       ///////////////////////////////

 if (inByte[12] > 0) {
        chord[0][2] = 81;
      } else {
        chord[0][2] = -1;
      }      

 if (inByte[13] > 0) {
        chord[1][2] = 79;
      } else {
        chord[1][2] = -1;
      }

 if (inByte[14] > 0) {
        chord[2][2] = 76;
      } else {
        chord[2][2] = -1;
      }

  if (inByte[15] > 0) {
        chord[3][2] = 74;
      } else {
        chord[3][2] = -1;
      }

   if (inByte[16] > 0) {
        chord[4][2] = 71;
      } else {
        chord[4][2] = -1;
      }

 if (inByte[17] > 0) {
        chord[5][2] = 69;
      } else {
        chord[5][2] = -1;
      }
      ////////////////////////////////////////

     if (inByte[18] > 0) {
        chord[0][3] = 71;
      } else {
        chord[0][3] = -1;
      }

    if (inByte[19] > 0) {
        chord[1][3] = 74;
      } else {
        chord[1][3] = -1;
      }

    if (inByte[20] > 0) {
        chord[2][3] = 76;
      } else {
        chord[2][3] = -1;
      }

   if (inByte[21] > 0) {
        chord[3][3] = 79;
      } else {
        chord[3][3] = -1;
      }

   if (inByte[22] > 0) {
        chord[4][3] = 81;
      } else {
        chord[4][3] = -1;
      }

   if (inByte[23] > 0) {
        chord[5][3] = 83;
      } else {
        chord[5][3] = -1;
      }   

    println("1: " + inByte[0] + "    " + inByte[1]+ "    " + inByte[2]+ "    "+inByte[3]+ "    "+inByte[4]+ "    "+inByte[5]);
    println("2: " + inByte[6] + "    " + inByte[7]+ "    " + inByte[8]+ "    "+inByte[9]+ "    "+inByte[10]+ "    "+inByte[11]);
    println("3: " + inByte[12] + "    " + inByte[13]+ "    " + inByte[14]+ "    "+inByte[15]+ "    "+inByte[16]+ "    "+inByte[17]);
    println("4: " + inByte[18] + "    " + inByte[19]+ "    " + inByte[20]+ "    "+inByte[21]+ "    "+inByte[22]+ "    "+inByte[23]);

    currentByte = 0; //after performed all these, current goes to zero
  }

}

FUTURE IMPROVEMENT

We would like to incorporate lights in the project so that  the light could indicate the pace of the music.
We are also interested in incorporate letters with the experience to allow people to deliver interesting messages while interacting in making the music based on pentatonic scale.

Comments are closed.