In a world full of interactive devices we find ourselves surrounded with sensors, joysticks, screens, etc. For this assignment, we decided to explore a different kind of input that does not require the user to press any buttons or screens, instead, we wanted to explore using the camera and user’s face to send commands to a computer. FaceTracking is a p5 application that uses the computer’s camera and an algorithm to read and understand the user’s face. This application is currently set to play a sound with the user’s face movement, however, any function can be added to this application.
In this project, we will explore using the user’s face, as a controller to send commands to the device. This tool is a basic prototype but has potential to be scaled to include any number of functions that run based on the user’s face manipulation and movement.
This tool is based on the FaceTracking application using Haar Detection technique, which uses an algorithm that contours the user’s eyes, nose, mouth, eyebrows, and chin. Each element of the face is given a number and then a vector is drawn connecting the numbers. Using this tool, we were able to make a simple beat player that the user can play simple music with.
High-Level Computer Vision focuses on a complex analysis of images. When talking about CV and faces, there are three major sections:
1) Detection: spotting the difference between a face and non-face,
2) Recognition: distinguishing different faces,
3) Tracking: a combination of detection and recognition over time.
We wanted to explore the face tracking option and create a controller using our faces. We started with Kyle McDonald’s Face Tracking Example.
We found this Class Notes from McDonald that explains all you need to know about CV and faces. OpenCV uses a Haar Detection technique, developed by Paul Viola and Michael Jones in 2001. Haar detection can be used for any two-dimensional object, but it can not handle any significant rotation or skew. It is also very limited on the colour variation that it requires. There is a video about HP Computers that could not follow Black faces.
The face tracking example identifies 70 points on the user’s face.
We took the key points that layout the face’s elements and draw contours around them. We didn’t notice a lot of change with the eyes, eyebrows, or the nose. But we were able to rotate the general contour of the shape. So we took the two points that defined the edges of the faces and compared them with each other. By comparing their Y position, we were able to identify if the face was tilting in any direction. After that, we added music to each direction so that the user would be allowed to play music by moving their heads.
Tools & Materials Used
P5.JS online editing tool
Two Mp3 files
Trying to determine which direction the face was angled either up or down was slightly confusing. We used a calculation to determine the exact point at where the controller would be activated but realized we did not need to after re-thinking our logic. (Omid can you elaborate on this part a bit? )
Future iterations could include a new beat everytime the user opens the page. So other could make a variety of beats together in front of the camera on their own devices. The webcam could also include more than one face and once again provide a random mix of different beat set per controller, since it currently only recognizes one.
As someone who experiments with musical instruments regularly, understanding tempo and rhythm are essential to playing any kind of music regardless of how simple. In big bands, because there are many musicians playing together, synchronization is essential and it is usually maintained using headphones for each member or a maestro to keep everyone on-tempo. Besides being unusable to people with some disabilities, headphones can be annoying and can cause unnecessary sweating around the ears. Following that logic, I wanted to create a tool that I can use during a jam session to replace the headphones, a tool that allows me to hear and interact with my surroundings while keeping the tempo.
This experiment is an attempt at using Arduino and vibration motors to create a wearable metronome. This metronome will use vibration instead of audio to notify the user of each beat, making it a more accessible tool to people with hearing disabilities on one hand, and others who have a preference of not wearing headphones. The goal is to create a p5 interface that gives the user the control of the tempo, but this version of the metronome only allows modification through hard-coding the Arduino board. So, it is important to note that this version produces one tempo at a time, but the intention is to continue expanding and aiming for full interactivity without the need to edit the Arduino code.
Class Experiments Through the three experiments, we were asked to take note of the findings and test different methods of connecting the circuit and of coding the motors. This was a very useful step because I was able to learn at an early stage what type of connection I am interested in having depending on the purpose. For example, connecting the motor to a transistor made the vibrations much stronger but required a more complicated circuit. On the other hand, using a motor driver offered aq library of vibration patterns but the intensity of the vibration decreased significantly. The second experiment offered me what I was looking for to build the metronome because it required a very simple reliable connection, allowed multiple motors, strong enough vibration and enough flexibility to edit the vibration patterns.
Body Placement Through my research for similar tools online, I came across the Soundbrenner Pulse, which is a smart wearable metronome that the user wears on his or her wrist, like a smartwatch. I also was considering the wrist as a placement for the metronome based on Clint Zeagler’s diagrams, suggesting that the wrist is a high sensitivity area. However, while the wrist metronome can work well for some instruments, percussions instruments, on the other hand, might not work well with a wrist metronome. So, after several trials in placing the motors on different parts of the skin, I eventually decided to place them on the back of the neck. This way, it will ensure there are no cables in the way, the vibration is not hindered by the instrument itself. In addition to that, as part of my tattoo education, I know that the epidermal skin thickness of the neck is thin compared to the rest of the body. Placing the motors on the back of the neck brings the whole body into the musical experience because of the vibration that is pulsating into the body through the skin, which is more intense than placing the motors on the arms.
Circuit & Code Going through the class experiments allowed me to see different approaches for the circuit. Considering that more than one vibrating motor is required to build this metronome, I wanted a simple circuit that is reliable and is not hindered the number of motors connected to the board. It is important at this level to write a code that is scalable because this is only the first version and future iterations will have more motors and more options available. Eventually, I used a simple connection, using the one side of the motors to two digital pins and the other to the ground pins.
The code I used for this project is based on the fade example from the Arduino library. However, after trying different settings and values, I noticed that vibration with a (50) delay was enough to simulate a percussion tap. In addition to that, for this version, I decided to use a 75 bpm (beats per minute), which is a mid-range tempo. But since I am working with the “delay” function, I have to calculate the gap between the beats rather than the speed of the beats. So, through some calculations: 75 bpm will require an 800 milliseconds delay.
Tools & Materials Used
Arduino Mega 2x vibration motors
The main challenge I faced in this experiment was figuring out how to code the gaps between the beats rather than the beats per minute measurement. It got really complicated when I added another motor and attempted to output different patterns on each at the same time. However, since I am using the delay function, it is causing some issues and I will be working on it more to straighten out those issues.
In addition to that, one of the main challenges during this assignment was writing the p5 code. Although it is something we have done before, after numerous attempts to make it work, for some reason, it is still not sending any information through the serial port. I will continue working on that.
In order for this tool to be fully useful, it requires an equation that converts tempo values on the front-end to delay in milliseconds on the back-end. This is crucial since the tempo has to be changeable by the user easily. Also, I will explore using more than two motors for the next iterations and testing how the experience can change to the user. In addition to that, I will explore adding some more functionality into the metronome, like offering the option of splitting the beat into 3 or 4 sub-beats.
As we spend more and more time on computers, especially laptops, whether for entertainment or work, the long and dreadful list of injuries and syndromes keep getting longer and more real. Maintaining a bad posture for too long can cause injuries that in some cases can be chronic and extremely painful. So, building on that logic, the need for creative solutions for workstations and workstyles to maximize productivity and minimize side-effects has never been higher. As a result of this, I decided to use assignment 4 as an opportunity to explore what I believe the right approach is and build a prototype of that tool based on my own requirements and expectations.
Eventually, I decided to create a wooden laptop stand that raises the laptop about 15-inches off the tabletop, which allows the user to raise their heads and keep their neck at a healthier angle. In addition to that, it is highly recommended to take regular breaks while working for long hours to stretch, or use a toilet, etc. So, in addition to improving the user’s posture, I also added 14 LEDs at the bottom of the stand to communicate to the user that their work session is done, and it is time to take a short break. Similar products exist in the market, however, this is a DIY solution that anyone can re-create at home easily and without the need to spend a lot of money.
The main goal for this assignment is to use an ambient information system that acts as a timer for work sessions. To achieve that, I added an LED strip of 14 LEDs that would communicate to the user, through colour, that the 1-hour work session is finished and it is time to take a break. I decided to follow Pousman et al’s guidelines for the ambient information system in order not to interfere with the user’s focus and get their attention when needed.
In addition to that, another main objective of this project is to make sure it can still be considered a DIY project that can be disseminated online and shared with makers around the world. To be a successful DIY project, in my opinion, it has to be scalable to fit people’s needs and capabilities.
First, I started by figuring out the dimensions of the stand. I decided to raise my laptop screen by 15-inch because that was sufficient for my desk to keep my neck at a healthy angle rather than keep bending it downwards, which the usual posture using a laptop. For the dimensions, I decided to go with a size that fits a 17-inch laptop screen, I own a 15-inch MacBook Pro, which is sufficient for most laptops available today.
I decided to use wood because durability is something I care about deeply as a designer and wood is both strong and durable. Also, this stand will be used on tabletops and since most tabletops are wooden, a wooden stand is more consistent and less distracting to look at while working.
I decided to create a 4-legged rectangular stand because the four legs ensured four fixed points which are needed when using a laptop. After some trial and error, I realized that for a laptop stand, it is much better to rely on four legs because with 3-legs, the laptop was not stable enough and when both hands are used, can become really unstable.
After building the stand for the laptop, I started preparing to install the LED strip at the bottom. At first, I was just going to glue the LED strip to the bottom of the stand, however, after some trial and error, I noticed that the LED light are too overpowering based on Pousman et al’s guidelines. So, I decided to build a base for the LED strip that tilts the LED light inwards and deflects the light from the user’s foreground to the background.
Coding this device was a real challenge for me because, until this assignment, I have not been able to fully back an LED strip. With the help of Adafruit’s NeoPixel library, I was able to define each LED in the strip and give it different values at different times. Originally, I wanted to blend the colours and have them slowly change over time, I could not figure it out and after long trials, I decided to use a different approach.
Using Illustrator, I identified a specific spectrum of colours that I want to use to communicate time. Then I divided the hour into 12 intervals and assigned a colour for each interval. Starting with greens then blues then magentas, the user can tell in which third of their work session they are in and roughly how long they have until it is time to break. In the code, each interval was assigned 300,000 ms duration and a colour and using the “millis“ function, the program switched from the first to the last colour over an hour period.
Tools & Materials Used
LED Strip – 14 LEDs
The biggest challenge I faced with this project was hacking the LEDs fully. My aim was to assign two colours for the beginning and end of the duration and have the Arduino program blend between those two colours. However, after constantly trying, I decided to give up on that idea and pursue something more tangible and achievable.
Critical Analysis & Future Steps
In general, I am happy with the results of this experiment considering the requirements and expectations I set at the beginning. However, the choppy colour change between the colours is distracting and does not meet my requirements for this tool. So, I plan to continue looking into more ways to control LED strips, and find more blending options. When I tried to use the Adafruit NeoPixel library, my attempts to repurpose it for this project did not work.
Also, I want to consider looking into covering the LED with an acrylic sheet or another material in order to hide the strip and smooth out the light.
Life in the city requires a constant chase against time. From long demanding hours at work to brief hours we spend relaxing with family and friends. This anxious chase, while it might boost productivity for some time, is not sustainable for the body of many people, like me. I find myself overwhelmed, whether it’s the crowding or noise, and I tend to regularly and physically isolate myself from my surroundings to meditate and slow my heart rate and eventually my anxiety down.
StressBracelet is a biosensor that uses galvanic skin response to measure the stress or anxiety level and responds with the lighting of 12 LEDs when stress levels are high. The user of this device would wear the bracelet and go about their day as usual, if the LEDs switch on, or if the user can feel their anxiety rising, they can take a break and do some meditation which would lower the stress and that would lower the moisture in the skin, which in turn switches the LEDs off.
The idea behind this bracelet is that since we, as city dwellers, are all prone to stress, there is a need for a device that silently communicates with us our level of stress is high, and if needed, that we should take a few minutes to breathe and take control of it. Using colourful LEDs added a positive spin to this device, on one hand, the user might be experiencing anxiety, but on the other hand, they will also see a colourful mixture of light, which is a less stressful approach to interact.
The aim of this experiment is to propose a tool that would help people get rid of their anxiety in public spaces and especially on their commute. The sensor has to be easily portable and accessible on-the-go, while at the same time discrete and personal in its interaction.
Design Process (Documentation)
The reason I decided to create this sensor in a bracelet is because it is important for this device to be handsfree for people to use while they are walking or commuting. I am also interested in bringing to the table the concept of electronic components as a wearable statement that speaks to a digital generation. So, starting from this logic, I began to think of how a bracelet like this might look. Chain bracelets are some of the most popular bracelets used worldwide, whether made of silver or gold, chain designs have been the dominant style for bracelets for a long time.
So, building on that, I decided to experiment with the idea of chaining LEDs create a chain that is then connected to a controller, which then adds a certain functionality of the chain. However, I quickly realized how difficult it, because of the number of wires required to connect the LEDs to the 5V and the ground. So, I decided to tweak my idea and create a different iteration of the bracelet with the LEDs attached to it, rather than hanging from it.
For the finger sensor, I decided to use aluminum foil attached to velcro and wrapped around two fingers, index and middle fingers. Although the foil is sufficient for this experiment, it is extremely fragile and cannot be constantly re-used. However, through experimenting with the sensor, I recorded the serial readings and noticed that they were extremely unstable, but I was able to get an idea of the range, between 0 and 13 in a normal state, and 15-28 when activated. The experiment I made used meditation as a way to push moisture in the hands (from sweating due to stress) down and blowing hot air into them to activate the sweat glands and record higher readings. Through moisture in the hands, the voltage on the serial print through the Arduino is either increase (with more moisture) or decreases (with less moisture). This allows us to record those readings and measure to some extent the amount of stress a person is going through.
To build the circuit I followed a similar approach like the one we covered in our last experiment. Connecting one of the sensors to a transistor that is also sending the sensor value to the A0 pin, and the other to the 5V pin on the Arduino. Also, linking the LEDs to digital pins on the Arduino and the ground to the ground. This circuit allows the Arduino to read the amount of voltage that passes through the human skin, which increases when the sweat glands are activated through stress, anxiety, working out, etc.
Once all the parts were assembled, I moved to the Arduino code. I started off from the provided code in the class presentation slides, which only gave us the ability to control one LED. So, I decided to add 11 more LEDs in order to achieve a sort of ambient effect as a result of the rise in the stress level. In the code, I assigned pins 13 to 2 (with the exception of 10 and 23) as LED pins and connected the positive side of the LEDs to the digital pins on the Arduino. Also, I knew I had to add the “digital.write” line of code for the 11 new LEDs .
The biggest challenge I faced during this experiment was figuring out how to make all the components fit into a wearable device. One of the problems is I only own an Arduino Uno and Arduino Mega, and both are way too large to be considered for wearables. However, having a smaller controller will make this task a lot easier. Also, my original idea of connecting the LEDs and the components in a chain link turned out to be much more complex than anticipated, which led me to change my design drastically.
I am considering other options of how to put this device together for a wearable experience.
During my research into different ways to make this device, I noticed that some people were able to build something very similar without the use of a controller. I am very interested in pursuing that option further and seeing where that would lead.
For this experiment, I used parts from my own collection, however, for my next iteration, I plan to purchase a few parts in order to tweak the device even more.
Learning how to build a biosensor opens up many doors for future explorations into this subject. We can build and customize our sensors to match our bodies.
Tattoo practicing can be extremely frustrating because it’s difficult to find people who are willing to volunteer their skin for beginner tattoo artists to practice on. So, many artists start with fake skin, which is a tattoo practice sheet made of silicon. A very popular starting point, which might not be very good because it has no indication on whether the student is doing it right or wrong. I’ve been to many tattoo conventions and most tattoo artists I met are self-taught, which is usually a very long process because of a shortage of volunteers and lack of direction. I remember when I was starting to use fake skin sheets for the first time, I had no idea whether the needle was going too deep or too shallow, which can be problematic because the whole point of the practice is to build the correct muscle memory.
So, I want to use this experiment as an opportunity to explore how to extend the tattoo practice sheet to a more reliable tool for new tattoo students. The sensor I created allows the students to tattoo on the practice sheets while the tool is indicating levels of pressure: green for “okay”, yellow for “a little too hard”, and red for “too much pressure”.
When the tattoo student is ready, they will hold the machine as they would normally and start tattooing. It is vital to know the correct angle to hold the tattoo machine at because tattoo ink is supposed to be pushed by the needle into the Dermis, which is the second layer of skin we have under the Epidermis (top layer of skin). If the ink is not pushed far enough in, it will not stick and eventually will be pushed out by the skin. However, if the ink is pushed too far deep in, past the Dermis, the ink will be spread out too much and ruin the quality of the piece. So, it is vital for the tattoo artist to train their hand muscles to memorize the movement and the angle, with continuous practice, accurate muscle memory is achievable.
So, after the tool is assembled, with all the layers in the right order, the green LED will be lit. As long as the machine is held in the right angle, and the pressure on the needle is not too high, the green light will indicate that the student is on the right track. However, once the yellow LED is lit, the student has a chance to fix the gesture back to the green zone. Red LED is an indication that the pressure is too high and most likely it would’ve caused some bleeding on human skin.
First, I started with filling the measurements sheets, which is great because I was able to test the different values of the conductive textiles we received in our kits. This allowed me to see the variations between the values “at rest” and “activated” before I start building my sensor.
I tested the different textiles to see which would work best for my sensor, and I realized that, out of the textiles in the kit, the Velostat worked best for this tool.
Next, I started sketching different variations of what that tool might look like. As a tattoo student, I have some insights into what I believe how this tool should be used and how it should look like to effectively perform its intended function. On the other hand, this is a speculative approach, and I was open, throughout the process, to changes and modifications to the concept.
In the initial sketches, I was planning to use a regular foamboard for the back support, with conductive fabric and Velostat sheet between the back and the front layers. When I built this prototype, I quickly realized that without proper support, this device will not be functional as it’s supposed to, because the layers will constantly move which will cause unstable values.
For my second prototype, I decided to use wood as back support with four pins on each corner to hold each layer down in its place. The reason is that wood is not conductive, and since the wooden pins will be holding down every layer, a conductive pin can cause shortcircuiting and/or inaccurate results.
Once I had this prototype completed, it was a matter of tweaking and fine-tuning the parts in order to get the intended results. So, moving to the LEDs, which are essential to this tool’s function, was relatively an easy process. I used the generic 2-pin LEDs in 3 colors, green, yellow and red.
Although the tool I created is an early prototype with a lot of wires showing, I made sure that it is stable and portable. However, for my second prototype, I am planning to create a shell for all the parts, and soldering all the parts together in order to make sure it will last over long use.
The resistor I used for this sensor was a 26k resistor. This was the highest resistance I had and so I had to work with it. Using the 26k resistor dropped the values on the serial enough for me to be able to work with effectively. For example, with the 26k resistor, the “at rest” value was between 800 and 840. Although that’s a little high pressure, tattooing does require some pressure and so I decided to make the “green” up to 889, and between 890 and 909 is for “yellow” and above 910 is for “red”.
After testing the sensor with a tattoo machine, some fine-tuning had to be done, but more or less the response through the LEDs was sufficient for me.
To build this sensor tool, I decided to work based off one of the example codes provided in the Arduino software. However, the original code only included 1 LED with different functionality than what I’m looking for, so, some changes had to be done to the code.
First, I added adjusted the code to remove the outputValue because I did not need the values to be mapped to 255, instead, I used the original 0-1023 range and split it into three levels, 0-840 for acceptable pressure and added an “if statement” to let the green LED value “HIGH” when the statement is true. Then, I added an “else if” statement for the yellow and red values 840-910, then 910 and above respectively. This way, the relevant LED lights when the relevant pressure value is reached.
Building this tool was a special experience. I remember using the tattoo practice sheets and thinking that they were not very effective and I remember wanting to do something about it. This experiment gave this chance and I am glad that I came out of it knowing more than going in.
Learning the different uses for the conductive sheets was a helpful exercise, this is my first real exploration into the conductive fabric world, but I can already see how much value it can add to our lives. However, I also learned that they are not easy to work with and troubleshooting can be also difficult since there are no parts to fix. For example, the values I received on the serial port did not match the values I got when I was filling out the sheet at the beginning. So, I had to revisit those values and see why they are different and how I can try to fix them. Eventually, I was able to map the values I received
I also learned how important resistors are when using conductive fabric with Arduino. When I was testing different resistors to see how the values change, I noticed that without the correct resistor, it’s almost impossible to receive accurate or stable values.
As I mentioned before, this is an early prototype, even though it functions well enough, in my opinion still looks rough. So, for my next iteration, I would like to finalize the look and the style of the device. Moreover, I want to consider more possibilities for the materials besides wood, perhaps consider a 3D printed base that can house all the electronics, while providing support to hold the silicon and the Velostat sheets.
Also, I want to purchase a full set of resistors and test out a few more options for more accurate values. If I can drop the “at rest” value using a higher resistor, it can allow me to fine-tune the sensor even more, which would eventually make the response more accurate to the pressure of the needle on the tattoo machine.
During the ideation process in class, I got 8 suggestions from the pile and out of the eight suggestions, the one that stood out for me was “Happy Hat Skiing”. The reason why I liked this idea was that I enjoyed picturing a happy ski hat, especially when it combines knitting and sewing with technology to get some interesting results. This approach of combining past and future skills and techniques is a personally interesting field for exploration.
In the beginning, my idea was to create a hat that predicts the wearer’s mood and reflects it through an LED in the “fluff” at the top of hat. However, I am personally not sold on the idea of technology with mood prediction because of the social impacts it would have. So, eventually, I decided to go with a knitted and sewed hat with LED light in the fluff ball and ON/OFF functions rather than mood prediction.
Knitting was very challenging at the beginning and my first 6 or so attempts failed. However, after repeating a few times, I started to understand knitting and it can be used. I considered the option of knitting the conductive thread through with the yarn, but I wasn’t sure where I wanted to place the thread yet and so I decided to sew the conductive thread in the texture once I’m done knitting. I started with two 20-knot batches of knitted textile, not perfect, but enough to create a rough prototype of the hat by sewing the two batches together with normal thread.
Then I sewed a conductive thread from the LEDs to the positive and negative lines of the tester, which is sewed to the edge of the hat. When the circuit is closed, it lights the LEDs, which are 2 generic yellow LEDs connected using conductive thread to the battery.
I still find challenging to work with textiles since it is a new exploration for me and is still hard for me to visualize, especially in the ideation stage. However, this experiment pushed me to explore my prototype through and learn more about knitting. I would also like to explore different types of techniques and material for knitting because I believe knitting can be a surprisingly useful solution depending on how it is used.
For the next step, I would like to find a better way to close the circuit, perhaps using a button or a sensor to light the LED. Also, I would like to focus more on creating a better prototype of the LED fluff ball because it can be used on its own, added to any hat, to light up through a button or an app in the future. An interesting approach to that would be to explore using felting to create fluff ball that lights under a certain temperature or with rain.
If done nicely, the fluff ball can be a nice addition to any ski or winter hat.