“The Soundtrack Of Their Life …” (Social Bodies Project)

*Download a more readable PDF version of this documentation here:
TheSoundtrackOfTheirLife_KenLeung.pdf

Introduction

As most of us move through the course of our day-to-day lives, we traverse a textured landscape of
emotions — joy, sadness, anger, fear, awe, anxiety, and elation (to name a few) — all triggered by the
plethora of things that happen to us, and the things we think about.

However, we often don’t express the richness of these emotional tapestries to our loved ones at
the end of the day — we tend to talk about the ‘significant events‘ that happened, instead of the
fleeting ‘little moments’ that may be just as interesting or important. Or we may feel that the
negative emotions are too ‘heavy‘ to talk about, focussing only on the ‘happy‘ things.

With The Soundtrack Of Their Life project , I sought to create a ‘tool for empathy’ that enables
two people to  ‘listen in‘ on the ongoing dips and spikes in eachother’s moods on during the
course of a day, while separated from eachother in space.

The project consists of two pendants, each equipped with galvanic skin response (GSR) sensors
to measure the arousal level (mood) of its wearer. Each pendant will send an ongoing stream
of GSR data to the other pendant (via an XBee radio), which is then abstracted into
music — a calm mood will elicit low, steady tones, while a more aroused state (anxiety,
intense happiness), will cause higher-pitched notes to play.

While it is recognized that biometric data may not be an entirely accurate measure for
something as complex and nuanced as human mood, the ultimate goal of the project is
to encourage a greater degree of dialogue and conversation between the two
wearers at the end of the day, by making them more cognizant of eachother’s
ongoing internal states (which are normally invisible).


A few scenarios may serve to illustrate this:

John:
I noticed that you were a little distressed around lunchtime today ..
mind if I ask you what happened?

Jane:
Oh, did my ‘music‘ seem to spike? No, quite the opposite — I just remember
feeling exhilarated as I was walking around — the sun was shining, the smell of
spring was in the air, and I suddenly had felt like everything was right in the world
and in my life, that I didn’t need anything else. Ever have moments like that?

—-

Mary: Your music seemed to be ‘peaking‘ all day .. are you ok?

Bob: Oh … I’ve been going through some anxiety at my job the last few weeks — I
haven’t been getting along with the new boss and it doesn’t seem to be getting any better …

Mary:  Oh no .. why didn’t you tell me about this?

Bob: I guess I should have .. didn’t want to burden you ..


Galvanic Skin Response

What is Galvanic Skin Response?

Galvanic Skin Response (GSR) is a measure of the electrical conductance of skin,
which varies with the degree of moisture secreted by the body’s sweat glands
(degree of moisture). Since the activity of sweat glands is controlled by the
sympathetic nervous system, (which responds to the body’s overall level of
stress and arousal), GSR readings are meaningfully correlated to a subject’s
emotional state. GSR sensors are traditionally employed in polygraph
(lie-detector) devices, and are also used in hypnotherapy sessions, to ensure
that a hypnotic state has been achieved.

How is GSR measured?

Galvanic skin response is measured by detecting the level of conductance between
two points on the skin, by sending a small amount of current is along the skin via
body-mounted electrodes. The conductance will be higher for individuals in a
high state of arousal (increased moisture level).

Related GSR-Based Wearable Projects

Prototype Physiological Sensing System,
MIT Media Lab (Affective Computing Group)
http://affect.media.mit.edu/areas.php?id=sensing

Shimmer
http://www.shimmer-research.com/

GER: Galvanic Extimacy Responder,
Kristin Neidlinger
http://www.fashioningtech.com/profiles/blogs/ger-galvanic-extimacy

Cold Feet – Interactive Wedding Bouquet
http://www.fashioningtech.com/profiles/blogs/cold-feet-interactive-wedding


GSR Sensor Experiments

I built two preliminary iterations of GSR Sensor hardware,
to ascertain the type of input data being read from the skin.


Iteration 1: Finger Loops As Electrodes

Materials:

Breadboard (1 pc)
Arduino FIO Board (1 pc)
10k Resistor (1 pc)
0.1F Capacitor (1 pc)
Insulated Wire (2 pcs)
Tin Foil, 1”x 2.5” (2 pcs)
Velcro, 1”x1” (2 pcs)

Method:

Two electrodes, which were each looped around a finger,
were fashioned using tin foil and velcro. The electrodes
were then wired into the Arduino FIO board (using sensor pin
1 to read the level of conductance). The 10k resistor and 0.1F
capacitor served to reduce ‘noise‘ in the circuit, yielding a more
accurate reading.

The design and code for the GSR circuit was adapted from the
following projects:

http://www.extremenxt.com/gsr.htm

http://cwwang.com/2008/04/13/gsr-reader/

Arduino code loaded onto the FIO read data from Sensor Pin 1,
which was then printed to the Serial Monitor.

Code:

// GSR sensor variables

int sensorPin = 1; // select the input pin for the GSR sensor
int sensorValue; // variable to store the value coming from the sensor

// Time variables

unsigned long time;
int secForGSR;
int curMillisForGSR;
int preMillisForGSR;

void setup() {

// Prepare serial port

Serial.begin(9600);
secForGSR = 1; // Change this for GSR
curMillisForGSR = 0;
preMillisForGSR = -1;

}

void loop() {

time = millis();

curMillisForGSR = time / (secForGSR * 1000);

if(curMillisForGSR != preMillisForGSR) {

// Read GSR sensor and send over Serial port

sensorValue = analogRead(sensorPin);

//Serial.print(sensorValue, BYTE);

Serial.print(sensorValue, DEC);
Serial.println(“”);
preMillisForGSR = curMillisForGSR;

}

}

Testing & Findings:

I tested the GSR sensor with 3 different users, to ascertain the stability of incoming data.
Readings were recorded for the users at:

i) Calm state (generally recorded at the beginning of the test)

ii) Anxious state (user was instructed to think about something stressful)

iii) State of pain (user was instructed to pinch themselves in the arm)

It was found that the data from the GSR was relatively stable — each user maintained a
reading of 21 during ‘calm state’. ‘Anxious’ and ‘painful’ states yielded a consistent pattern
of increase (a higher reading for anxiety, and an even higher reading for pain), although
the exact readings themselves varied between users.

Iteration 2: Thumb Tacks as Electrodes

For iteration 2, I simply used thumb tacks as electrodes instead of the foil-based fingerloops.
The reason for this: to allow for the possibility, in the final design of the wearable,for electrodes
that don’t need to be strapped to a body part, but resting against it instead.The thumbtack sensors
were tested by pressing a finger against each — this yielded nearly identical patterns of data to the
Finger Loop method.

XBee Communication

The two XBee modules were configured via AT commands (so that they would send data to eachother),
using Tom Igoe’s XBee Radio Terminal app for Processing, and an XBee Explorer dongle.

Communication between the two XBees was tested using the following apps for
sending and receiving (adapted from Tom Igoe’s ‘Xbee Analog Duplex Sender’ code,
from Making Things Work

Sending Code:

// Sends a number to destination XBee

int destId=2;
int count=0;

void setup() {

Serial.begin(9600);

}

void loop() {

count++;
Serial.print(count, DEC);
Serial.print(“\r”);
delay(1000); }

Receiving Code:

// Receives a number from destination XBee

int inByte=-1;
char inString[6];
int stringPos=0;
int count=0;

void setup() {

Serial.begin(9600);

}
void loop() {

if (Serial.available() > 0) {

inByte=Serial.read();
inString[stringPos] = inByte;
stringPos++;

if(inByte==’\r’) {

int number = atoi(inString);
Serial.print(number);
Serial.println(“”);

for (int c = 0; c < stringPos; c++) {

inString[c] = 0;

}

stringPos = 0;

}

}

}

Form & Body Placement

While brainstorming the most meaningful form and body placement
for the wearables, I progressed through the following versions:

1. Glove (Hand):

Each user would wear a glove, with the Arduino/XBee components embedded into the top, and
the pair of GSR sensors attached to the index and second fingers. Custom-made, plush
‘headphones‘ would extend from the glove to the user’s ears. This initial form sprang
directly from my original, finger-mounted iteration of the GSR sensor.

However, it didn’t make much sense from a social perspective — why would one’s hand
be a reader and conveyer of mood? Also, the rather exposed (and zany) nature of a glove
detracted from the more intimate and private experience of mood sharing that I was
trying to achieve — it would
prompt curiosity from onlookers.

2. Stethoscope (Neck):

For my next concept, I sought to place the wearable in a more meaningful place in
relation to mood and emotion — close to the heart. In this version, the GSR sensor
would rest on the user’s chest, with custom headphones extending upward, the
overall form resembling that of a stethoscope.However, this form brought with
it overly clinical connotations (examining, vs listening), and the headphones were
still too ‘showy‘ and couldn’t put away when not in use.

3. Pendant (Neck)

This concept took the form of a pendant (containing the sensors, Arduino, and
Xbee) which would hang around the user’s neck, with store-bought (vs custom) earbuds attached.

All exposed components of the pendant (the string, the earbuds) were commonplace enough
as to not draw attention from others, and the earbuds themselves lent a familiar connotation
of music listening.

Overall, I felt that this concept and body placement best conveyed the theme of the project —
listening to secret mood music from the heart of someone close to you — and was the
foundation for the final prototype.

The Final Prototype


Pendant: Phase 1

Materials (For Each Pendant):

Electronic Components:

Prototyping Board (1 pc)
Arduino FIO Board (1 pc)
XBee Series 1 (1 pc)
10k Resistor (1 pc)
0.1F Capacitor (1 pc)
Insulated Wire (2 pcs)
Store-bought earbuds (1 pair)

Non-Electronic Components:

Thumb Tacks (2 pcs, serving as electrodes)
Red plush fabric (2 pcs)
Black string
1/4” Foamcore

Construction:

  1. The Arduino, XBee, 10k resistor, capacitor, thumb tacks, and earbuds were soldered together
    on a piece of prototyping board (which was in turn mounted to foamcore,
    for stability, and to provide an attachment surface for the thumb tack electrodes).
  2. A red fabric pouch was made to house the components of each pendant,
    and a pair of thumb tacks were affixed to the back of each (so that they contacted
    the skin of the chest when worn).
  3. Black string was attached to each pouch so that it could hung around the neck


Arduino Code:

The final Arduino programming combined the XBee send/receive code
and GSR data processing code from initial experimentation, together
with music-playing code which translated GSR readings into sound frequencies.

GSR data was multiplied by 10 before it was ‘sonified’, in order to bring it into
a frequency range which featured a meaningful amount of pitch variation
(highs and lows).

Code:

// GSR sensor variables

int sensorPin = 1; // select the input pin for the GSR sensor
int sensorValue; // variable to store the value coming from the sensor
int songPin = 8;

// Xbee variables

int inByte=-1;
char inString[6];
int stringPos=0;

// Time variables

unsigned long time;
int secForGSR;
int curMillisForGSR;
int preMillisForGSR;

void setup() {

Serial.begin(9600);
secForGSR = 1; // Change this for GSR
curMillisForGSR = 0;
preMillisForGSR = -1;
pinMode(songPin, OUTPUT);

}

void loop() {

// Begin Serial Read

if (Serial.available() > 0) {

// Read serial data from Xbee and build temporary string

inByte=Serial.read();
inString[stringPos] = inByte;
stringPos++;
// CR is delimiter marking the end of one piece of GSR data

if(inByte==’\r’) {
int number = atoi(inString); // convert string to int
beep(songPin,(number*10),500); // play frequency based on GSR value

// Clear the temp string variable

for (int c = 0; c < stringPos; c++) {
inString[c] = 0;

}

stringPos = 0;

}

}

// End Serial Read

time = millis();
curMillisForGSR = time / (secForGSR * 1000);
if(curMillisForGSR != preMillisForGSR) {
// Read GSR sensor and send to destination XBee

sensorValue = analogRead(sensorPin);
Serial.print(sensorValue, DEC);
Serial.print(“\r”); // Add CR as delimiter (end of GSR data string)
preMillisForGSR = curMillisForGSR;

}

}

// Beep code by Leah Buechley

void beep (unsigned char speakerPin, int frequencyInHertz, long timeInMilliseconds)

{

int x;
long delayAmount = (long)(1000000/frequencyInHertz);
long loopTime = (long)((timeInMilliseconds*1000)/(delayAmount*2));

for (x=0;x<loopTime;x++)

{

digitalWrite(speakerPin,HIGH);
delayMicroseconds(delayAmount);
digitalWrite(speakerPin,LOW);
delayMicroseconds(delayAmount);

}
} // end beep


Findings And Challenges:

  1. Lack of conductivity between electrodes and skinDuring initial tests of the chest-placed pendant, there seemed to be a lack of conductivity
    from the electrodes (thumb tacks). No sound was being emitted, pointing to an uncompleted circuit.

    At first, I thought the problem might be the lower density of sweat glands in the chest area
    (vs the fingers). However, it turned out that an issue of insufficient contact area between
    the electrodes and the skin. While the thumb tacks worked well as electrodes for the soft
    and pliable tissues of the fingers (which conformed to the thumb tacks to form a good
    electrical connection), the hard surface of the breastbone was less yielding.

  2. Aesthetic form of the pendant did not imply ‘sending‘ The visual form of the pendants implied entirely individual entities — it would be ideal
    if there was a stronger implication of the communication between the pendants,
    as ‘social’ devices.


Pendant: Phase 2


Based on the challenges faced in Phase 1, the following modifications were made:

  1. Change of location/ materials for electrodes Instead of thumb tacks, strips of tin foil were used as electrodes (much larger
    conduction area). These new electrodes were then mounted on the ‘string’ of the
    pendant itself, so that they rested on the back of the neck when worn — this would
    ensure a better connection due to the action of gravity pulling down upon them.
    The string was substituted for electrical wire, to connect the electrodes back to the
    Arduino, and heatshrink tubing was applied to the wire to make them more nondescript.
  2. Two colours for the pendants New enclosures were made for the pendants, with a different colour for each (red, or green).
    A circular piece of the ‘opposite’ colour was then affixed to each pouch, to imply a connection
    between the two.

Next Steps

  1. Multiple Biometric Sensors A larger array of biometric sensors would be useful to
    create a more nuanced reading of ‘mood’, such as:

    – Heart rate sensor
    – Respiration sensor
    – Brain wave sensor

  2. Increase the communication range of the two pendants
    The two pendants currently communicate with eachother via their
    built-in Xbees, which have a range of only up to 100 feet. To fully achieve
    the project’s goal, the pendants should be able to talk to eachother over
    greater distance (e.g. between the wearers‘ office locations in different
    parts of the city). The next step would be to investigate XBee-to-Internet
    communication, via the ConnectPort gateway.
  3. Reduce Size of the Device
    Due to the arrangement of internal electronic components, the current pendants
    approached 2” in thickness, making them bulge outwards from underneath clothing
    (instead of being inconspicuous as originally intended). Future iterations should be
    more compact.
  4. Jewellery-like material for enclosure
    Later iterations should be enclosed in a solid material such as wood or clay, which is more jewellery-like (also implying a ‘precious‘ connection between two people).
  5. Headphone jack for the earbuds
    A jack embedded in the pendant, enabling the wearer to plug in their own earphones,
    would provide a stronger connotation of a ‘music-listening device’.
  6. More nuanced sound
    The current Arduino code generates 8-bit sound which is heavily reminiscent of retro video games, putting a light and humorous cast upon all the generated mood music.Using more advanced sound synthesis might yield more ‘mood-appropriate‘ music,
    with a wider range of emotional ‘highs and lows’.

Leave a Reply

A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.