PoseNet Experimentations

Workshop 6 Notes
GitHub

PoseNet Experimentations: Replacing the mouse with different tracked body points, and using gesture as a type of controller through PoseNet js.

Concept

PoseNet js is a position tracking software that uses real-time data analysis to track 17 key points on your body. The key points are translated into a javascript object which documents at what pixel position each point is currently at on the webpage. These experiments attempt to see if these pixel positions can replace the computer mouse functionality and create new control gestures through tracked body movement.

Process

I introduced myself to PoseNet through Daniel Shiffman’s coding train example that uses PoseNet with p5.js. This example was thorough and explored the base concepts of:

  1. What does PoseNet track?
  2. How do you extrapolate that data?
  3. How can you use the key data points analyzed through computer vision in relation to another?

A clip of myself following Daniel Shiffman’s PoseNet example. Note how the code is written to isolate certain keypoints in “keypoints[1]” etc. 

PoseNet uses computer vision to document the following points in the form of “pixels” on your body with a declared accuracy. The library for PoseNet allows you to identify the poses of multiple people in a webcam. My explorations will be focussing on actions and gestures from one participant.

A list of the key points on the body that PoseNet analyzes and creates positions for in relation to the video.
A list of the key points on the body that PoseNet analyzes and creates positions for in relation to the video.

This data is then translated into JSON that the PoseNet library instantiates as a data structure and that beginning of your code.

A screenshot of the JSON that PoseNet provides that documents the coordinates.
A screenshot of the JSON that PoseNet provides that documents the coordinates.

The JSON gives you an x and y coordinates that indicate the position of each point being tracked. Because of the way the JSON is structured you are always able to accurately grab points to use as inputs.

I found that the points are very similar to p5.js mouseX and mouseY coordinates. I decided to pursue a series of small experiments to see how to use gestures as a mouse movement on the screen.

Screen position & interaction

I wanted to see how I could use the space of the screen to interact with my body that was analyzed for data points on a webcam. I wanted to explore raising my hand and see if I could activate an event within a part of the screen. Interestingly enough, PoseNet does not track the point of a user’s hand but is able to track their wrist. I decided to experiment with the wrist point as it was the closest to the hand gesture I wanted to explore.

I used the method in Shiffman’s tutorial to isolate my wrist (key point 10). I then divided the webcam screen into 4 quadrants. I found the x and y coordinates of my right wrist. If the coordinates of my right wrist were located within the top left* quadrant, an ellipse would appear on my wrist.

*Note, the webcam image is a flipped image of the user. The right hand is shown on the left side of the screen.

Video documenting myself activating the top left quadrant with my hand. 

I expanded on this idea by seeing how I could use this gesture as a controller to stop and start an action.  I chose to draw an ellipse centred on my nose. Upon page load, the colour of the ellipse would randomly be chosen. If the coordinates of my right wrist were located within the top left quadrant, the colour of my nose ellipse would start to change.

Changing the colour of my nose by using the gesture action of raising my right wrist to randomly chose the red, green, and blue values. 

When the wrist leaves the quadrant, the nose takes on the last colour that was randomly chosen. This was a quick proof of concept. The loop happens quite quickly so that the user has little control over colour is selected.

The right wrist point was an interesting key point to chose. I found myself always tracking my hand rather than my wrist. Upon testing, I noticed myself having to raise my hand higher than I anticipated to activate the quadrant. I do not think this is because of the tracking accuracy, but rather the human discrepancy vs the machine understanding of where does my wrist start and end.

2) Buttons (almost)

I was able to isolate a specific quadrant on the page and could start and stop actions. I was curious to see if I could emulate a button click, and give further didactic information through HTML inputs.  

I created a button that would change the colour of the nose upon hover. I found the coordinates of the button, the width and height, and then compared this to the value of the nose pose. For this experiment, I chose the nose rather than the wrist because I wanted a one to one relationship with the control of the action and the gesture (rather than having a different body part control the output of another point).

As per the previous experiment, p5.js generated a random colour for the nose. When the nose interacted with the button on the page that declared “change colour of nose”, the colour of the nose would change colour.

PoseNet tracking my nose and interacting with a button created in p5.js

This button is an artificial stand-in since there is no mouse press event that is fired. It is simply tracking through position. I considered attempting to feign a mouse press, but chose to not because of the outcome of this design was the same desired result.

3) Mapping

My next experiment went to further gesture through the exploration of mapping. I found the nose ellipse colour to be an effective representation of action on the page. I chose to map the coordinates of my hand across the screen to the colour of the nose ellipse.

A screenshot showing the coordinates of the right elbow.
A screenshot showing the coordinates of the right elbow.

A video of my wrist controlling the colour of my nose through a sweeping gesture. 

I enjoyed how this emulated a slider. P5.js has the capability to include a slider for volumes and other values, such as RGB. This gesture almost replaces the functionality. The next step for this gesture would be to “lock” the value of the slider rather than just having a continuously mapped value.

4) Two body points interacting together

For this experiment, I wanted to explore two parts: the first, how was it possible to create an “interaction” with two points; the second, is it possible to trigger an output like a sound?

To achieve this, I chose to see if I could use my knee and my elbow to create a gong sound when they “collided” together.

I started off by finding the coordinates for the knee and the elbow. The x and y coordinates provide very specific points, it is not a range, nor is there an identifiable threshold that is given. These two sets of coordinates are not tied to the screen as the previous experiments and would need to respond in relation to each other.  This posed a bit of an issue, how would I determine if the knee and elbow were colliding into each other? It would be too difficult and unintuitive to attempt to have both the knee and elbow interact when they had the exact same coordinates. I decided that the knee would need to surpass the y coordinate of the elbow marginally to indicate a connection between the two points.

Two start the visual representation of this experiment, I did not draw anything on the screen. When the knee and the elbow “interacted”, an ellipse on the knee and elbow would draw.

A video of my knee and elbow attempting to trigger the drawing of two ellipses.

I found that this was effective every other attempt. I added ten extra pixels in each direction to see if this would help expand the range. This helped only marginally.

My next step was coordinating sound. I found a free gong sound file that I loaded into p5. This, unfortunately, did not work as I wanted to. Since I was triggering a sound through the .play() function, when my knee and elbow collided for more than a 1/60 of a second, the .play() function was executed. This was the beginning of a journey of callbacks and booleans that I decided not to include as it was deterring me from the exploration of PoseNet.

The knee and the elbow colliding was not as satisfying as an action or of use as a controller as I would have liked. The action itself is very enjoyable and having the sound of a gong would turn the body into an instrument. This action would work better if the sound was continuously playing, and then body controlled the volume or other aspects of the song rather than controlling the “starting” and “stopping” of the audio.

Conclusions

PoseNet provides a useful data model that allows people to located their key body points on a screen. The points are accurate coordinates that can be used to interact with the screen without having to use a mouse. The experiments I conducted are initial explorations into how it is possible to use the data from the PoseNet as inputs. The explorations can be used as basic models for common mapping functions, mouse clicks, and multiple input events. The output of the colour changing nose can be replaced with outputs such as sliders, changing filters, or starting and stopping events. PoseNet is a powerful tool that can take complex physical data and create a virtual map of coordinates for people to interact with online.

My next steps in these explorations would be to explore more complex and combined patterns of interaction. All of these explorations are single actions that provide one result. PoseNet offers the ability to use up to seventeen points of data from one person, which can be multiplied with every person interacting on the same screen. The multiple poses could lead to interesting group cooperation exercises, such as all participants raising their left wrist to create an output. The basic experiments of input and output still apply, though the logic would need to change to account for multiple bodies.

Overall, gesture-based controllers seem futuristic but the technology has become extremely accessible through PoseNet and built-in webcams. PoseNet allows an easy introduction to browser-based computer vision applications. These experiments offer a very basic understanding and introduction into common gesture interactions that now become possible through PoseNet.

Research 

PoseNet documentation

How to Build a Gesture Based Game Controller 

Real-Time Estimation Using PoseNet

Haptic Slider

Haptic Slider
Workshop Notes #5
Olivia Prior

Demo of the Haptic Slider
Demo of the Haptic Slider

Concept
Haptic Slider is an experiment that explores movement, response, and time through the use of soft sensors. The slider measures the motion and position of a finger along a sewn path. As the finger moves along the path from left to right, the intensity of the vibration motor increases accordingly. The haptic response creates a dynamic experience; instead of engaging with a dial to control a vibration the user can feel and set the intensity to their liking through the initial contact with the path. The finger becomes the controller and mimics user experience patterns learned on touch devices. The Haptic Slider takes digital patterns that create depth in devices without any and implements the patterns on physical tactile surfaces to give the user a sense of control.

Objective
For this project, I was inspired by common user experience patterns that are learned through touch devices like our smartphones. Many of those user experience patterns create space on surfaces where there is visually nothing more than a screen. I wanted to explore how to apply these motions we have learned (such as sliding a volume control down) and apply them onto a surface where depth cannot be changed visually. My experiment is to explore what could this haptic response create when the user performs the action of sliding a finger up and down on a soft surface.

Process
The first instinctual action that I wanted to perform was the act of a finger moving up and down a soft static slider. I drew out a brief sketch of what I wanted to achieve.

Initial sketch of the Haptic Slider.
Initial sketch of the Haptic Slider.

Upon looking at my sketch I realized that what I really wanted was a physical potentiometer. I went to How to Get What You Want and explored the tutorials. I found a fabric potentiometer that used resistive material, conductive fabric, and conductive thread.

Screenshot from How to Get What You want of the diagram for how to construct the soft potentiometer.
Screenshot from How to Get What You want of the diagram for how to construct the soft potentiometer.

To follow the tutorial, I decided to change shape of the slider pattern to match the design that they had created. I chose to use Eeonyx Pressure Sensing Fabric resistive material that is used in a pressure sensor as an experiment.

The inner circle is conductive fabric and the outer ring is resistive material.
The inner circle is conductive fabric and the outer ring is resistive material.

I chose to sew these pieces onto a piece of non-conductive fabric for stability. I used alligator clips with male headers to connect the two pieces of material to my breadboard. I then took a small piece of scrap conductive fabric and moved it along the dome-shaped path of the soft potentiometer.

I uploaded the example code analogInOutSerial to read the differences in values.

The soft potentiometer connected to power and ground
The soft potentiometer connected to power and ground.

Initially, I was receiving a reading between zero to three. I swapped out my 10k resistor for a 1k and found that my range ranged from thirty to three hundred as my piece of fabric moved along the edges. Satisfied with the outcome, I decided to connect a vibration motor to the circuit.

Vibration motor connected to power and ground using alligator clips with male headers.
Vibration motor connected to power and ground using alligator clips with male headers.

I initially tried to map the value of the sensor to the output value of the vibration motor. I found this ineffective because the motor only really triggered a vibration when the output value was greater than 150. Because I wanted a steady gradient of vibration as someone’s finger travelled along the path I chose to not use this method.
I uploaded the example code for the Haptic Controller Breakout Board. The example code cycles through all of the patterns that are available in the library. I held the vibration motor until I felt a gradient of steady vibrations go from either high to low (or vice versa). The serial port was printing out the number of the example as it was executing the associated code. After a couple of loops, I identified some key pattern sequences that created a steady decrease in vibration. In addition, on the documentation for the breakout board is included a pattern list. I found the specific pattern numbers and isolated the code to loop through those.

Screenshot from the Adafruit documentation of the settings I used for the vibration motor.
Screenshot from the Adafruit documentation of the settings I used for the vibration motor.

My next step was connecting the potentiometer and the pattern library. Through my initial testing, I found that the potentiometer rarely exceeded 300, and the lowest value was 0 when there was nothing touching the fabric.

Mapped values from the output to the vibration motor.
Mapped values from the output to the vibration motor.

I created case statements that would determine the increase or decrease the intensity of the motor depending on the analog value from the fabric.

Case statements to determine the gradual increase or decrease in the virbration motor intensity.
Case statements to determine the gradual increase or decrease in the vibration motor intensity.

I found that this was simple and effective, but I had an issue when the value was not consistently zero. If it was not consistently zero, the motor would often “glitch” on and off.

After testing again, the initial value of touching the sensor at the farthest edge was thirty, so I decided that if the output value was below twenty-five to set the value to zero.

Arduino code that sets the value to 0 if the incoming reading is below 25. This is to stop the vibration motor from flickering.
Arduino code that sets the value to 0 if the incoming reading is below 25. This is to stop the vibration motor from flickering.

For documentation purposes, I included an led light on the circuit to visualize the motion of the someone tracing their finger along the path.

Soft circuit all assembled. An LED light is included in the circuit using conductive fabric to give visual feedback for documentation.
Soft circuit all assembled. An LED light is included in the circuit using conductive fabric to give visual feedback for documentation.

I sewed a finger cover out of conductive fabric rather than fumbling with a piece of scrap conductive fabric.

Finger slip made out of conductive material that is used to interact with the potentiometer.
Finger slip made out of conductive material that is used to interact with the potentiometer.

Video demonstration of the Haptic Slider 

Connection Diagram 

Connection diagram for haptic slider.
Connection diagram for haptic slider.

Results & Next Steps
The potentiometer gives accurate haptic feedback that correlates to the position of interaction on the path. The interaction is a very one to one result. Once the user takes away their finger all elements of interaction immediately stop. The result is no longer a tactile controller but a response to the interaction the user is engaging within. Wherever the user places their finger will dictate the intensity of the vibration.

In the next iteration, I would want to make the slider have a memory and work more like a tool. The user could slide their finger along the path to dictate the level of vibration and once removed the vibration would stay at the last indicated intensity.

I really enjoyed the design as the shape mimics a gauge, similar to that of a speedometer. The visual connotations work well with the response and haptics of a vibration motor. In this design, the vibration motor is visibly separated from the potentiometer and the sewn together base material. To create a more authentic haptic experience, the vibration motor would be placed beneath the potentiometer to give a hidden haptic response. Currently, the visual of the vibration motor is an initial indicator of the intensity as the user interacts, but ideally the user would choose an interact with the tool solely using touch. The other visual indicator that is in this circuit is the LED. This could be removed in future iterations. Currently, it is included for as a visual reference for documentation.

The user is required to use a piece of conductive material to connect the potentiometer pieces together. On How to Get What You Want, there is a demonstration of a potentiometer that uses metal pieces on top of conductive and resistive fabrics. I was tempted to explore this tutorial but I disliked the concept of having to move something physical on top of the potentiometer for this iteration. Ultimately though, it is necessary to connect the two pieces and I chose to sew a finger covering to attain the “magic” of the finger being what controlled the potentiometer.

Conclusion
Overall, I really enjoyed the simplicity and delicacy of crafting the soft potentiometer. Haptics is an interesting exploration because it borders what is simply a reaction and what is a reaction that gives the user meaning to their action. In common haptics we interact with, we receive information on depth and pressure. In this exploration, I aimed to see if I could explore both of these. I am unsure if the one to one relationship that resulted in the interaction of moving a finger along a fabric potentiometer achieved either fully. The mapped interaction of moving the finger along the path to a vibration intensity could have used in practical products as a controller, but on its own, it is simply a calculated response. In a future iteration, it would be interesting to experiment with how this design could be embedded in other technologies to further explore the field of haptic response in body-centric technologies.

References

Wake Up Curtains

Timer Curtains
Workshop #4 Notes
Olivia Prior

Demo of the wakeup curtain pattern.
Demo of the wakeup curtain pattern.

Concept
Timer Curtains are curtains that use thermochromic paint to allow light into a room by gradually removing the colour from the pigment on the patterns of the fabric by using electrical voltage.  The heat from the voltage turns the painted pattern to white which allows for more light to enter the room. The curtains are activated by a timer or an alarm that allows the user to set either a time to wake up or to set a time that they would like to get up and take a break. The patterns gradually remove themselves one by one, so the transition from patterned curtain to white is an ambient experience. This allows for either a slow wake-up or slow reminder to that a break is coming up.

Objective
With this project, I wanted to create a product that used patterns as a form of time telling. Upon exploring the thermochromic paints I was curious as to what would happen if you made multiple tiny patterns and could control them individually. I had painted dots onto a sheet and it reminded me of the UI/UX pattern of “loading ellipses” you often see on websites. I wondered how you would be able to use this known UI/UX pattern into the physical world as a notion of timekeeping, but also keeping the connotations of a moment “loading”. I wanted to see if it was possible to gradually remove the dots over time to create a white sheet, to explore the applications of this method, and to see if it was an effective use of a clock.

Workshop notes & ideation write up
Workshop notes & ideation write up

Process & Explorations
I started off by painting patterns using two colours of thermochromic paint. I initially wanted to test how far the heat from the voltage could spread along one colour before bleeding into the adjacent colour pattern.

The patterns painted with thermochromatic pigment.
The patterns painted with thermochromic pigment.

I created multiple different patterns on many sheets of white fabric, and once dry sewed some resistive thread through the different colours.

Resistive thread stitched along the length of one of the pigment colours.
Resistive thread stitched along the length of one of the pigment colours.

I attempted to run 5V power from my Arduino to the sewn thread using alligator clips that were simply connected to power and ground. At first, I saw no change. After seeking some advice, I realized that the distance of the sewn circuit was a big factor in the amount of heat that would be dispersed through the thread. The longer the circuit with a lower voltage, such as 5V, would only be seen if the circuit itself was small.

I adjusted the placement of the alligator clips to the first few stitches and immediately saw a difference.

Demo of heat from the circuit changing the pigment on one small part of the circuit.
Demo of heat from the circuit changing the pigment on one small part of the circuit.
Back view, showing the alligator clips attached to the resistive thread. The heat from the current has changed the colour of the pigment form the back.
Back view, showing the alligator clips attached to the resistive thread. The heat from the current has changed the colour of the pigment from the back.

I tested to see if I could make the thermochromic paint fade in an out be connecting the power to a digital output pin on the Arduino. Then I made the delay twenty seconds between turning the power on and off on the pin. In the video below you can see a slight change in the pigment.

Video if thermochromic pigment changing from the heat of the circuit. 

With this knowledge and the set of tools that I had, I assessed the pieces that I painted. I saw that the dots each had a small surface area and came across the idea of slowly fading out the dots individually as a form of an ambient technology. The dots could be used as a timer: the user could set an amount of time, and the dots would slowly fade out. The dots once “vanished”, would indicate that the time is up without interrupting any thought. In terms of installment, I envisioned this pattern on windows curtains. As the dots turned white, they may gradually let in more light into the room and giving the illusion that the person who set the timer should go outside for a break. If the changing of pigment let more light into the room, the curtains would extend the act of ambient notification from simple pattern change to a lighting change in the space.

Envisioning the dots on a full length curtain. There are currently around 60 dots on the curtain, to symbolize minutes.
Envisioning the dots on a full-length curtain. There are currently around 60 dots on the curtain, to symbolize minutes.

Testing
I sewed small stitches along the length of each dot to start. I then clipped an alligator clip onto the power and ground of each dot. I made sure that the dangling threads did not touch each other as I was aware of the small working space I had created for myself. Reflecting upon the soft circuits project, I did not want to have a clean workspace.

Close up detail of the resistive thread sewn into the dots.
Close up detail of the resistive thread sewn into the dots.

I then moved each of the pins into the Arduino digital pins and ground. I made my code turn on a new pin every five seconds. The change of the pigment was gradual but effective. Nearly the entire dot had disappeared.

The dots changing colour from the circuit.
The dots changing colour from the heat of the circuit.
Screenshot of simple code used mock up the dots fading in sequential time.
Screenshot of simple code used mock up the dots fading in sequential time.

Diagramming for future iterations
I started to consider how I would be able to expand this project further and to place emphasis on the concept of a clock. I sketched out some process ideas of how this could operate and look:

Envisioning the dots on a full length curtain. There are currently around 60 dots on the curtain, to symbolize minutes.
Envisioning the dots on a full length curtain. There are currently around 60 dots on the curtain, to symbolize minutes.
Envisioning the dot curtain in action. The dots slowly disappear in correlation to the time that was set for them.
Envisioning the dot curtain in action. The dots slowly disappear in correlation to the time that was set for them.
As the timer nears the end, the dots have nearly all faded.
As the timer nears the end, the dots have nearly all faded.
Once the timer finished, the dots would all be completely faded.
Once the timer finished, the dots would all be completely faded.
This pattern envisions the dots in a clock like formation, to very physically represent time.
This pattern envisions the dots in a clock like formation, to very physically represent time.
This curtain envisions what "would happen if the curtain was entirely covered in pigment?" The curtain is striped and the thread would be sewn down the patterns to slowly fade out the stripes.
This curtain envisions what “would happen if the curtain was entirely covered in pigment?” The curtain is striped and the thread would be sewn down the patterns to slowly fade out the stripes.

In these images, I had included two colours of pigment on each dot. Aesthetically speaking, I really enjoy the complimenting colour contrast of both pigments. In my first trial (the “rainbow” pattern), the bleed of the heat from the voltage did not breach onto the second colour due to the width of the brush stroke. Having different colours fade in and out at different times could be used as different types of timers or indicators.

Other use cases for the combination of thermochromic pigment and curtains.

  • Inspired by Harry Potter, the curtains could be used as a “remembrall”. The remembrall is a device that changes state when the owner needs to be reminded of an item of importance. 

Scene snippet of character Neville Longbottom using a “Remembrall”

  • Wake up curtains. The curtains could be given a time and aid in an ambient wake up, similar to the Philip’s Hue Lights wake up on alarm function.
Two images of the Philip's Hue Lights interface for the wake up and go to bed "routines"
Two images of the Philip’s Hue Lights interface for the wake-up and go to bed “routines”
Two images of the Philip's Hue Lights interface for the wake up and go to bed "routines"
Two images of the Philip’s Hue Lights interface for the wake-up and go to bed “routines”
  • Stopwatch: the opposite of a timer. Once set, the pattern could fade in rather than fade out. The pattern could reflect time passing, rather than counting down.
  • The different colours could reflect different weather outside. If the temperature is hotter, the blue could fade away, or vice versa with the pink hues.

Questions to consider for future iterations

  • How would one interface with the curtains? Would there be a web app that they could use to indicate the timer? A button, potentiometer, and LCD screen combination for changing the time, or starting, and stopping the process?
  • How dense does the pattern need to be to affect the light in the space? If the curtains do not affect the lighting of the space, does the fabric need to be hung like a curtain, or could it be a wall hanging?
  • How could the wiring be improved for the patterns? The length of the connections affects the heat generated from the circuit, which would ultimately affect the fading of the pattern. Would using conductive thread and having short connections to multiple power sources be the most effective?  
  • Would having the curtains alter the user’s position in the room? What if their work station was turned away

Conclusion
 The thermochromic pigment is a very accessible and enjoyable technology to explore within soft electronics. The resistive fabric when connected to a circuit has a very similar behaviour to a switch, which when connected to either bespoke interfaces or APIs allows for endless possibilities of applications. Through exploring the concept of time and curtains as an ambient technology I discovered that power management and circuit management are the most concerning factors in setting up a viable product. These mini-experiments are explorations into looking at the possibilities of how one could add thermochromic pigment as apart of their electronics toolkit. Overall the pigment is a useful tool that has lots of applications for creating bespoke experiences embedded in everyday environments.

References and research 

Kobo Kant write up on thermochromic paint

Example of using thermochromic paint and resistive thread on “soft’ materials in a circuit 

Remembrall

Philip’s hue lights ambient wake up tutorial

Compact Tones

Workshop #3: Soma Aesthetics & Bio-Feedback
Compact Tones
Olivia Prior 

Concept

Compact Tones is a compact mirror that responds to the user’s stress levels. The mirror gently hums when left alone and is activated by being held by the user’s touch. The mirror responds to the moisture on the user’s palms and correlates this to the tone emitting tone. The tone changes every five seconds unless the user is in extreme duress. If this is the case, then the tone will repeatedly alter until placed down. This device gives space and reflection through audio to bring attention and presence to the act of gazing at oneself in a mirror for an extended period.

Objective

With this assignment, my goal was the create a self-reflexive experience on adjusting and looking at yourself in front of a mirror. I wanted to create a relationship with the mirror that is a non-visual experience to reflect on the act of engaging in an activity that was purely visual. The use of the compact mirror is to reflect the one on one relationship that we have with our own bodies. This project aims to highlight the act of habitually monitoring our own bodies and how one may use appearance as a coping mechanism when they are stressed.

The range of the tones was selected to avoid connotations of irritation when the tone is too high or apathy if the tone is not present. The mirror is always gently humming, and the pitch does not exceed too high of a tone. This does not dictate or prescribe any standard of a norm as the tone is always changing slightly when by itself. When the mirror detects the user’s touch it alters to a chirp, as if it were having a conversation with the user. This allows the user to be in full control of the device and their part in the conversation, and motivation in using the device.

Workshop notes
Workshop notes

Process

For this project, I chose to use galvanic skin response as a method of bio-feedback. I chose this method because I wanted to measure the activity of someone holding a contact mirror.

My first step was to test out the technique myself. I took two pieces of aluminum foil and taped them around my fingers to see if I could get a measure. I found at first that I was not getting a drastic change in numbers, but the more I tried I noticed my readings were increasing. This was an interesting experience as I was visually seeing the change through the readings as I was constructing the piece.

Initial investigation of the sensor.
Initial investigation of the sensor.

My next step was to visualize how the aluminum foil would be placed on the mirror. As I needed two detached pieces of aluminum I decided to place the foil on either side of the packet. For the speaker, I decided to place it on the bottom of the compact so that the emitting tones would not be as blaring.

Diagram of the compact mirror being held.
Diagram of the compact mirror being held.
Top view of the compact mirror assembly.
Top view of the compact mirror assembly.
Bottom view of the compact mirror assembly.
Bottom view of the compact mirror assembly.

After testing and sketching, I started to assemble my piece together. I cut out two pieces of aluminum foil roughly the size of each side of the compact mirror. I used electrical tape the secure the pieces down. I left two slots open on either side to allow for the alligator clips to clip into easily.

Materials required for assembly of the sensor.
Materials required for assembly of the sensor.

Video of the assembly of the compact mirror

I tested out the values with the compact. Similar to my first experience, I found that my readings were not significant until after a few times when my hands were actively moving and producing sweat.

I then connected the speaker to the circuit and added in tones. Initially, I created an array with a list of tones that were in the order of a musical scale from low C to a high C. I did this so that I would have any obnoxious tones that would be distracting to the experience of the mirror. I then took the values of the compact mirror and mapped them to the length of the array. The outcome of the mapping would be an integer that would then be used to indicate what value of the array would be the emitting tone.

Initially, the range of tones was distracting. When untouched, the mirror would emit a very low buzz, and then when activated the pitch would starkly change to a high whining tone.  I then mapped the tones to go from a low to a mid-range pitch so that the interaction would be more musical and smooth.

My next step was to include a time aspect to the piece. I did not want the compact mirror to instantly respond to palm because that would make the compact mirror more or a musical instrument rather than a tool for slower reflection.  I chose to code the compact mirror to take into account the sensor reading in correlation to the time spent holding the mirror. For testing purposes, I chose five seconds as a time measure. If the sensor value is over a certain threshold, and a certain amount of time passes (five seconds) then the tone scale would increase as a response. Once put down the tone would then go back to a lower tone.

Arduino code for the compact mirror and GSR sensor
Arduino code for the compact mirror and GSR sensor

Demo of the compact mirror responding to being touched.

Something I discovered as I was testing was that if the compact mirror was left by itself untouched, the tone would change every five-second increment. This was interesting and I chose to leave this as it made the experience of the tone changing when picking up the device a more natural and less judgemental interaction.

Another discovery I noticed when testing was that once I removed my hand from the device the tones would “wind down” in the scale in quick succession. Once again, it seemed like it heightened the natural interaction of human to mirror and decided to leave the code untouched.

Challenges & Outcomes

The main challenge of this project was thinking of how to create a non-judgemental response when holding a compact mirror. Having a reaction to looking yourself in the mirror could be a very loaded action: it could be immediately judgemental, it could be stressful, or it could a deterrent from the action. My initial thought was to have LED lights to respond to the proximity of being in front of a mirror, but I thought that the light would interfere too much with the act of looking at your reflection. I ended up playing around with tone as an alternative but was also concerned with the connotations of tones. Such as: Would it sound too much like an alarm? Would it turn into an instrument rather than create a reflective space? These were my main concerns pursuing this output.

I found that I did not a steady data set to work with using the aluminum foil. I thought that I would have a more gradual input of data, but it seemed fairly binary as I was working with the material. I also suspected the readings to be closer to 0 when the mirror was left alone but found that it was steadily around 200. Rather than calibrating the data set, I chose the threshold for the sensor to reflect a very high level of moisture on the skin (900 out of 1023), and for the lower inputs to be mapped to the tones so there was always the “hum”.

I found that using a found object to be an effective foundation for this project. Placing the aluminum foil on the mirror and using alligator clips to connect it the breadboard worked well for testing. Beyond testing, this would be inefficient for a portable item. If I were to make a future rendition having all of the components together would be problematic. The aluminum foil on the compact mirror is flimsy and only attached by the edges. I found that the foil ripped easily when the mirror was moved around extensively. As well, the foil covers all of the surfaces. This would be problematic for attaching any microprocessors or protoboards as they would most likely to be attached to one of those surfaces. The components would either have to be wireless or the foil would have to be connected to accommodate to the hardware.

In a future iteration, I would use recorded sounds rather than tones. I think the tones are playful but still come across as distracting and stark. If there were softer sounds that were less removed from the context of alarms the soma aesthetics of the interaction would give more space for reflection.

Overall, I think that this project could be elevated to be used beyond compact mirrors. The inputs from the galvanic skin response sensors mapped to sounds could be applied onto many other items. The act of physical touch mapped to an ephemeral output such as sound creates space for reflection on many other interactions such as using a pen or typing on a keyboard. Galvanic skin response as bio-feedback is interesting because it can incorporate time – the data inputs can change over the time of the interaction. This project is only a small experiment in the greater potential of using the galvanic skin response in combination with sound as an output.

References and Research 

Galvanic Skin Response Tutorial 

Basic Tones Code for Speaker Output

Galvanic Skin Response Explained 

Hydrator

Workshop #2 Notes
Olivia Prior

Github for the Hydrator

Context

This assignment directed us to test and get the readings from different types of conductive materials. We then took one of the materials and created a bespoke sensor that measured and calibrated to a body related action. I chose to create a sensor that measures how much water someone has consumed from their water bottle.

Testing the materials with the multimeter

For the first step of this project, I and a few classmates measured the different materials in our conductive fabric kits using a multimeter. We tried to be as clinical as possible with the materials and let the material rest naturally after it had been activated. The only material we did not do that for was the fibre because we wanted to see the contrast between the fibres being spread out versus tightly scrunched together.

Material: Velostat #1
Description: Matte plastic sheet
What activates it: Pressure (or bend)
How to connect: Secure conductive material to front & back face

Round # Value at rest Value activated Multimeter Resis. setting Observations
1 140000 ohm 126 ohm 200k (rest) – 200 (activated) This material seemed to have the resting values extremely high vs. the activated material. The sensor did not need a terrible amount of pressure to change.
2 32000 ohm 130 ohm 200k (rest) – 200 (activated)
3 35000 ohm 220 ohm 200k (rest) – 200 (activated)

 

Material: Velostat #2
Description: Shiny plastic sheet
What activates it: Pressure (or bend)
How to connect: Secure conductive material to the front & back face

Round # Value at rest Value activated Multimeter Resis. setting Observations
1 44400 ohm 120 ohm 200k (rest) – 200 (activated) Similar results as before, but this material seemed to have a lower resting value than the other Velostat. The sensor did not need a terrible amount of pressure to change to a lower resistance.
2 44800 ohm 109 ohm 200k (rest) – 200 (activated)
3 40300 ohm 102 ohm 200k (rest) – 200 (activated)

 

Material: Eeonyx Pressure Sensing Fabric
Description: Coated woven textile
What activates it: Pressure (or bend)
How to connect: Secure conductive material to the front & back face

Round # Value at rest Value activated Multimeter Resis. setting Observations
1 50000 ohm 150 ohm 200k (rest) – 200 (activated) A large range of values! Found to be a bit inconsistent with the resting values.
2 39000 ohm 72 ohm 20k (rest) – 2000 (activated)
3 33100 ohm 133 ohm 20k (rest) – 2000 (activated)

 

Material: Eeonyx Stretch Sensing Fabric
Description: Stretchy knit textile
What activates it: Stretching Fabric
How to connect: Clip power and ground to either end

Round # Value at rest Value activated Multimeter Resis. setting Observations
1 145000 ohm 53900 ohm 200k resistance Did not need to change the resistance for this material; the range seemed to be fairly consistent.
2 140000 ohm 53400 ohm 200k resistance
3 134500 ohm 50900 ohm 200k resistance

 

Material: Eeonyx StaTex Conductive Fiber
Description: Fluffy fibres, similar to cotton stuffing
What activates it: Squishing/compression
How to connect: Clip power and ground to either end

Round # Value at rest Value activated Multimeter Resis. setting Observations
1 12500 ohm 690 ohm 20k (rest) – 2000 (activated) This one was fun trying to scrunch it to as small as it could be. The range is quite large for this material as well. This material is incredibly responsive; a light touch changes the values significantly.
2 12300 ohm 450 ohm 20k (rest) – 2000 (activated)
3 13100 ohm 550 ohm 20k (rest) – 2000 (activated)

 

  1. For testing the Arduino’s we followed a similar clinical process: test the materials and let them rest naturally. Upon initial investigation, I was convinced that we had our setup incorrectly because the values upon activation were going up rather than lowering. After reflecting we realized that this made sense because the closer the sensors (especially the pressure ones) were together the closer the path is for the current.

Material: Velostat #1
Description: Matte plastic sheet
What activates it: Pressure (or bend)
How to connect: Secure conductive material to the front & back face

Round # Value at rest Value activated Resistor Value Observations
1 20 1015 4.7k Will not go higher than 1023? Even with different resistors. The value would often go down to nearly zero. Inconsistent resting values. The sensor did not need a terrible amount of pressure to change.
2 3 1015 4.7k
3 40 1013 4.7k

 

Material: Velostat #2
Description: Shiny plastic sheet
What activates it: Pressure (or bend)
How to connect: Secure conductive material to the front & back face

Round # Value at rest Value activated Resistor Value Observations
1 95 1021 10k Same as above, will not go higher than 1023? Even with different resistors.

The sensor did not need a terrible amount of pressure to change.

2 95 1022 10k
3 93 1022 10k

 

Material: Eeonyx Pressure Sensing Fabric
Description: Coated woven textile
What activates it: Pressure (or bend)
How to connect: Secure conductive material to the front & back face

Round # Value at rest Value activated Resistor Value Observations
1 50 1021 10k Will not go higher than 1023? Base rate is lower than the velostat
2 60 1022 10k
3 70 1022 10k

 

Material: Eeonyx Stretch Sensing Fabric
Description: Stretchy knit textile
What activates it: Stretching Fabric
How to connect: Clip power and ground to either end

Round # Value at rest Value activated Resistor Value Observations
1 54 118 10k The resting values were inconsistent. The values of the rest vs. activated were always about half.
2 60 122 10k
3 62 128 10k

 

Material: Eeonyx StaTex Conductive Fiber
Description: Fluffy fibres, similar to cotton stuffing
What activates it: Squishing/compression
How to connect: Clip power and ground to either end

Round # Value at rest Value activated Resistor Value Observations
1 570 980 10k A large array of values! I scrunched this one very tightly. Just over double the value?
2 588 976 10k
3 544 950 10k

Creating the Hydrator

For this assignment, I chose to measure the act of consuming water. My goal was to fabricate a sensor that was connected to a water bottle that would:

  1. Calibrate the sensor to the pressure from the water in the bottle
  2. Indicate and prompt the user that they should be drinking through the use of a timer
  3. After the user has consumed some water and the water bottle is placed back onto the table, re-calibrate the high and low values of the sensor
    • If the value is the same, the user did not drink water and do not reset the timer drinking timer
  4. Restart the timer and indicate to the user that they have consumed water.

I chose to fabricate the pressure sensor out of the matte velostat, conductive fabric, and the floor foam. I chose those materials as I wanted to create a shape that takes an even reading from the base of the water bottle. There were lots of this material to take from, so I knew that I would be able to produce the size and shape to take the reading I wanted.

My first step was to trace the base of my water bottle and cut out the shape of the sensor. The bottom of my bottle dipped in slightly at the middle, so I chose to use a “donut” shaped sensor that would only attach itself to the bottle’s points of contact with the table. I cut the conductive fabric slightly smaller and included flaps for the hardware connections to have enough surface area to attach.

Traced out cutouts from the base of my water bottle.
Traced out cutouts from the base of my water bottle.
Cut outs of the conductive fabric and velostat.
Cutouts of the conductive fabric and velostat.

My next step was to cut out the foam pieces and assemble the sensor. I cut the foam to be a bit wider than the velostat to allow for full coverage of the sensor. I was concerned that, because I was working with a device that carries water, there may be some potential for water to seep into the sensor.  

Assembling the water bottle sensor.
Assembling the water bottle sensor.

Before I fully attached the sensor together, I tested the values. To my surprise, my values were vastly different from my Arduino testing. My range was from 10-250, rather than 0-1023. This worried me at first, but I chose to continue with the development of the sensor and see what I would be able to do with the incoming data.

I attempted using different resistors to test the incoming data values. I found I had the best range using a 10k Ohm resistor. This was the same resistor that provided the best results in our initial testings.

Testing out the sensor before assembling it together with an adhesive.
Testing out the sensor before assembling it together with an adhesive.

After confirming that I was able to get a reading from my sensor, I used electrical tape to secure my sensor together.

Sensor attached together with tape, and connected to the circuit with alligator clips.
Sensor attached together with tape, and connected to the circuit with alligator clips.
Sensor secured together with electrical tape to avoid any drips of water from the bottle.
Sensor secured together with electrical tape to avoid any drips of water from the bottle.

My next steps were to attach the sensor to the water bottle and take readings with different values of water.

The water/pressure sensor beside a list of sensor readings that are taken in 100ml intervals.
The water/pressure sensor beside a list of sensor readings that are taken in 100ml intervals.
Sensor attached to the bottle.
Sensor attached to the bottle.

I found that there was not a significant range between 100ml intervals. This was an interesting challenge; I had initially planned to take readings on how much a user had sipped. The only major data differences were between 200ml intervals. I found that this was an unrealistic amount for a person to drink in one sitting each time the bottle would prompt them to drink.

Due to the data not providing a large enough range, I chose to include time as an indicator. If the bottle was “picked up” the sensor would not have any weight on it, therefore we could assume that the user would be drinking. The time the bottle was picked up would be timed to help ensure that the person was actually consuming.

I wrote out my entire process before I started coding.

The code process written out to help understand the order in which to write the code.
The coding process written out to help understand the order in which to write the code.
Fritzing diagram showing the LED and sensor connections.
Fritzing diagram showing the LED and sensor connections.

My first step was to ensure that the program would know that the bottle was being picked up to consume water. I tested this by using a strip of addressable LED lights as an actuator. If the bottle was unattended for more than 3 seconds, the LED lights would change to red. If the bottle was picked up the LED lights would change green. When the bottle was placed back down the timer would start again.

My second step was to calibrate the values of the sensor. I wanted to incorporate the ability to measure how much water was in the bottle after the bottle was picked up. For the first five seconds of the code, the sensor would calibrate.

I coded the rest of the program based off of these calibrated values. If, when prompted with a red LED light, the bottle was picked up for more than three seconds, placed down, and the sensor value was lower than the original calibrated sensor it would register that the person drank enough water. Otherwise, the LED light would stay red indicating that the water consumption quotient was not satisfied.

The functionality of this aspect was difficult to achieve. The results of recalibrating the sensor were inefficient. I attempted to pour out more water in an attempt to create a drastic difference in sensor values, but the values were still so minor in comparison. I chose to stick with the minimal viable functionality of lifting the water bottle up when prompted with the LED light and placing it back down again after three seconds as an indication that I drank enough water.

Results

Overall, I found this project challenging as the different sensors that I perceived to all be the same were actually drastically different. It was helpful, though, to figure out the intricacies of the sensor by creating a datasheet, as I had for the different millimetres. This whole process creates a relationship between you and the sensor and ultimately becomes an intimate experience. The sensor was crafted by your own hand and the sensor has its own unique data set.

In the future, I think I would choose to use a tilt sensor to indicate if the act of water is being consumed. I think that having the tilt sensor connect with two pieces of perpendicular conductive fabric would achieve that same concept, without the finicky aspect of the data set.

Even though the data set was much smaller than I had expected and less sensitive to the difference in the volume of water, I was surprised by how responsive the sensors were to light touches. When I was initially testing my readings for the water bottle, I could get a nice varying set of data by drumming my fingers along the top of the sensor. This could be applied in many different sound-based or visualization works.

Overall, the sensor I crafted allowed me to create a bespoke drinking experience: a bottle that responded through LED feedback to the frequency of intervals you were consuming water, and for the timed length you were in the act of drinking water.

Research & References 

Kobakant – Simple Fabric Pressure Sensors

Kate Hartman – Into to Textile Game Controllers

Run Adorably Sock

Workshop #1 Notes

“Run Adorably Sock”

By: Olivia Prior

Strategy + Goal

My prompt for this project was to create a run adorably sock-. I knew that initially I would not be able to fabricate an entire sock. Instead, to start, I wanted to create an interactive attachment that could be placed onto a sock. The attachment would be activated by the press of the user’s heel down onto the ground in order to track the wearer’s running movement. For the adverb side of the experiment, I wanted to create a cute cat face on the outside of the ankle that would light up when the button was pressed. Both together would create a bespoke attachment that lets the sock wearer run adorably.

Figure 1: Initial sketch of my prompt "Run Adorably Sock"
Figure 1: Initial sketch of my prompt “Run Adorably Sock”

From the experiments in class, I chose two different techniques. First, the knitted button and second felting and conductive fibres. The knitted button made the most sense to me as I essentially wanted an open circuit that closed when the heel pressed down. I chose the felting for the sturdiness of the material. If this was a device that was placed on the bottom of the foot, I needed the circuit to be sturdy and bound securely into the attachment material. I chose to execute the “adorably” part of the prompt by using un-felted fibres as the whiskers of the cat to connect the circuit.

Documentation

My original idea was to create a circuit that looped around the entire length of the foot, along the sole to the top of the foot and back past the heel. I initially traced my entire foot onto felt, but as I started to felt my fibres into the material I realized I had not fully thought through the connections. Looping around the entire foot would use a lot of excess material that was not required. Upon this realization, I took apart the felted fibres from the foot and drew some more thorough diagrams of my connections.

Figure 2: Full sole of the foot with the start of the circuit felted through to the tops of the toes.
Figure 2: Full sole of the foot with the start of the circuit felted through to the tops of the toes.

I diagramed my connections to wrap around the heel and ankle of the foot. I then crafted paper sensors to understand how I needed to attach all of my sensors together.

Figure 3: Bottom view of diagram, layer 1
Figure 3: Bottom view of the sock, layer 1 showing the felted circuit
Figure 3.2: Bottom view of the sock, layer 2 showing the first piece of knitted material.
Figure 4: Bottom view of the sock, layer 2 showing the first piece of knitted material.
Figure 5: Bottom view of the sock, layer 3 showing the first piece of knitted material with conductive thread .
Figure 5: Bottom view of the sock, layer 3 showing the first piece of knitted material with conductive thread.

 

Figure 6: Heel/Back view of the sock, showing felted circuit.
Figure 6: Heel/Back view of the sock, showing felted circuit.
Figure 7: Side 1 view of the sock, showing the "adorable" cat with the LED eyes and conductive fibre whiskers
Figure 7: Side 1 view of the sock, showing the “adorable” cat with the LED eyes and conductive fibre whiskers
Figure 8: Side 2 showing the battery attached to the sock.
Figure 8: Side 2 showing the battery attached to the sock.

The only part that I had less certainty about were both of the LED lights for the cat eyes. I chose to experiment and include them in this experimentation for my own personal research.

As I took apart the fibres on my first iteration of the full sole. I noticed that there were lots of stray fibres poking through. I was going to reuse that specific piece of felt but I was unsure if the stray fibres would contaminate my new circuit.

Figure 9: The back side of my initial iteration with lots of small fibres spread throughout the felt.
Figure 9: The back side of my initial iteration with lots of small fibres spread throughout the piece of felt.

I started to tear the fibres into pieces and then felt into the new heel attachment following my circuit diagram. The button would complete the gap in the circuit. The felted perpendicular lines were designed to give more surface area for the button to touch.

Figure: Felted the circuit to only the heel of the attachment.
Figure 10: Felted the circuit to only the heel of the attachment.

I took a break from felting and knitted my button sensors. I was excited that they were able to work just by pressing the pieces together. In my design, I did not need three knitted pieces, but rather two. I was going to use the felted circuit as the connecting part for the button. As I was knitting the conductive thread into the piece, I left gaps at the top and bottom of the knitted square because I wanted to be cautious of shorting my circuit. I was aware that the space on the heel I had allotted for the button was small and I did not want to have any risk of the button connecting to the other side of the heel circuit.

Figure 10: Two knitted pieces for the button; the first without conductive thread and the second with.
Figure 11: Two knitted pieces for the button; the first without conductive thread and the second with.

I tested the button using my testing tool to ensure that my loops were large enough for the knitted in the conductive thread on the first piece. To my surprise and contentment, it worked.

Video 1: Testing my knitted button using my testing tool 

I went back to felting and making the adorable cat with LED eyes. This process was challenging and frustrating. I had felt a piece of material in between the eyes to allow the first LED to end and the second LED to begin. Using the conductive fibres was challenging for this part; I had chosen them as the connections for the LEDs because I thought there would be a thick enough base for the wires of be able to stick into. I had also thought that I would be able to felt over top of the connecting part of the LEDs to secure the connection. In the end, I only was able to make the first LED light up briefly and could not confidently repeat the result.

Figure 11: The connections in the back of my cat were close together making for a messy circuit.
Figure 12: The connections in the back of my cat were close together making for a messy circuit.

I attached the cat face to the circuit and tested it using my testing tool. I was able to get satisfactory results of using the whiskers to “brush” along the felted circuit to pass the current through.

Video 2: Testing the fibres in my cat to see if they complete the circuit

I then sewed my knitted button onto my circuit. This I found challenging as I was not getting the bright consistent results I had with my testing tool. I placed my testing tool on either side of the button and was getting dim results. The electricity was passing through, but my connection was not strong enough. I believe this was due to my design of the large gap. In a future iteration, I would need to sew the knitted piece that contained the conductive thread to the circuit to enforce a stronger connection.

Video 3: Testing the knitted button sewn into the circuit.

The end result: my piece could be clipped to the base of the foot and to the ankle to act as a sock that helps someone run adorably.

Figure 13: Full circuit of the attachment
Figure 13: Full circuit of the attachment

Insights:

This assignment felt very similar to my first encounters with jumper wires and a breadboard. The first time I was following along in Creation and Computation my circuits were not colour coded or organized. As I was going through that assignment, I was thinking about how in the future I would be much more intentional with my stitches and keeping a clean working space. This is reflected in the way I aim to set up my circuits now; I am very intentional with ensuring I can see everything that is happening. I found the best example of frustration caused by straying from this technique when felting the fibres. Initially, I was felting away not really considering how the fibres on the other end would interact. As I started testing, I consistently went back and ensured that my working space was clean. This was a huge hassle and in the future I would use felt to block out certain parts of the felting area to ensure a contamination free zone.

I also re-learned the importance of sketching out my circuit and pursuing the project with intention. I think the connotations of the materials lends itself to a crafting mindset. It is hard to re-wire those instinctual urges such as; I will cut away the felt; or, I can use glue to repair that later. What helped the most was making my components out of paper first and then laying them down to understand my prototype. This allowed me to consider dimensions, connections, and what materials I would need to use.

Through my development, I found myself asking the question: “how would one actually wear this?” rather than “how would one construct this?” My first concern was about construction, but as I was developing the circuit I found myself placing the attachment on my foot and rethinking my design for wearable use. In my next project, I think this will be one of the questions in the forefront of my mind: how do I include the body aspect rather than just focus on the functionality?

Information sources:

I followed the tutorials that were offered in the class: the knitting tutorial and the felting tutorial for constructing my circuits. I looked at the website How to Get What You Want to view some LED light attachments for inspiration but I chose to experiment with connecting to straight into the felted fibres instead.

Next Steps:

  • Include buttonholes and a sock to attach the piece together.
  • Consider using conductive fabric or thread to construct the circuit rather than felting, as felting is messy in narrow spaces.
  • One could knit an entire sock with this circuit in it, rather than making an attachment.
  • Fully attach a battery to the circuit rather than a quick stitch for security.
  • Make a matching pair for the sock.