Machine Learning: PoseNet

By: April and Randa
Link to sketch code: https://editor.p5js.org/aprildezen/sketches/gkVoUOBeo

Strategy
To explore and understand how PoseNet works and how it can be altered and changed. We don’t have a lot of coding experience so we decided to take this slow and really understand the tutorial by Daniel Shiffman. Once we understood that we wanted to see what the limitations were and what we could build using this platform.

screen-shot-2019-03-09-at-2-56-00-pm

Documentation
Following the tutorial, we went step by step to create the canvas and set up the video capture. Before moving on we played with a few of the filters provided in the P5 library.

screen-shot-2019-03-09-at-2-17-46-pm

Once that was set up we moved on to setting up PoseNet. By using TensorFlow.js PoseNet model, the computer pulls up poses that match ours in real time, and displays skeleton-like graphics of our body. PoseNet can be used to estimate either a single pose or multiple poses, but just for simplicity we experimented with the single pose that detects only one person in a video.

We copied into the p5.js editor’s index.html file the below url for the ml5.js library: <scriptsrc=https://unpkg.com/ml5@0.1.3/dist/ml5.min.js”type=“text/javascript></script>

Importing this library allows the program to detect and trace keypoints positions in the face and body using the webcam. Then one can create interactive art or graphics that respond to body and face movement.

All keypoints are indexed by part id. The parts and their ids are:
screen-shot-2019-03-10-at-7-37-41-pm

When it detects a person, it returns a pose with a confidence score and an array of keypoints indexed by part id, each with a score and position.

screen-shot-2019-03-10-at-7-37-50-pm

This information allowed us to identify the positions of each pose keypoints, and also the confidence score of detecting them. After retrieving this information we played around with the code and tried out assigning shapes to different positions and keypoints. We learned also that to make shapes scalable we need to input their size to be a variable defined by the distance between the nose and an eye.

screen-shot-2019-03-09-at-2-25-30-pm

This is another example we explored that creates wobbly eyes.

screen-shot-2019-03-09-at-2-28-35-pm

Insights
We were both really blow away with how accurate PoseNet’s location detection was. We both played with it alone on our computers in class and it was eire how other it was right.

However, There is a lower accuracy detecting keypoints of the body with multiple people then there is with just one person. We played around with a few demos and noticed the lag time increase when we were both in view. In the video above, the clown nose was juggling between both our noses, trying to detect the more confident output. It actually was kinda a fun effect, we felt it felt like playing digital ping pong with our noses.

Now that we played with the demos, time to see what we can do with it.

WELCOME TO OUR ALIEN ABDUCTION.

screen-shot-2019-03-09-at-2-33-05-pm

Unfortunately, we forgot to hit record on this part because we had a lot of fun trying to figure out how target other points in the PoseNet library and add silly alien anteni and understand how the example code for the googly eyes worked.

Here is the breakdown of what we went through to become extraterrestrials.

Step 1: Locate where the ears are
Using the example code from the hour of code tutorial and the index of part id’s, we could easily locate the ear location – Left Ear 3 and Right Ear 4. As a placeholder we drew two ellipses over the ears to make sure it was working.

Step 2: Create anteni above our ear location
Now that we had red ellipses floating on top our ears, we needed to figure out a way to move them up above the ears. Since we just wanted to move them up, we only needed to change the Y axis. In the string that indicates the placement of the ellipses over the location of the ear, we made one slight change. We multiplied the the ‘ear1Y’ variable by .5 and that gave us enough height.

ear(ear1X, ear1Y*0.5);
ear(ear2X, ear2Y*0.5);

To create the anteni we simply change the x and y height so that the ellipse would we long and skinny and change the red colour to a RGB value that will give us bright green.

function ear(x, y, size, n) {
fill(164, 244, 66);
noStroke();
ellipse(x, y, 5, 100);
}

Step 3: Bring in the googly eyes
The googly eyes was part of the demo code that was included in the hour of code tutorial we watched on PoseNet. All we changed for this was the frame count to 2 so the eyes would spin faster and we changed the eye colour to green.

function eye(x, y, size, n) {
let angle = frameCount * 2;
fill(255);
noStroke();
ellipse(x, y, size, size);

fill(164, 244, 66);
noStroke();
ellipse(x+cos(angle*n)*size/5, y+sin(angle*n)*size/5, size/2, size/2);
}

Step 4: Add a filter to the video capture
The last thing we did to achieve this strange alien look was to add a filter to the draw function.

filter (INVERT);

Final Obduction #OOTD

screen-shot-2019-03-10-at-7-16-48-pm

Information sources
https://p5js.org/learn/color.html
https://p5js.org/examples/dom-video-canvas.html
https://p5js.org/reference/#/p5/filter

Reflection
Overall, we had a lot of fun playing with this. There is so much more we can do with it and though this learning experience we noticed how achievable it was for us to use it. Neither of us know much about javascript but we were able to figure out how it worked and how to start making changes. We think it would be interesting to continue exploring this tool.

Kids Haptic Set

By: April De Zen

screen-shot-2019-02-27-at-7-56-09-pm

Idea
I wanted to create a fun haptic set for kids. This prototype would act as a button to trigger an electronic toy or another wireless device. There will be 3 feedback points on this haptic set, 2 LEDs on each finger band and a vibration motor in the wrist band. When Botton is pressed the LEDs with light with a light vibration to indicate that the signal was sent.

Materials & Tools
1x old kids sock
1-3x vibration motors
3x sequin LEDs
1 roll of conductive thread
1 roll of regular thread
1 felt battery holder
3V round battery
Sewing needles
2x Alligator clips
Scissors

screen-shot-2019-02-27-at-7-58-23-pm

Workshop and experiments
In class, we tested out Motor Arrays and Haptic Motor Drivers. There were three experiments that allowed us to see the limitations and possibilities of a simple vibration motor. The biggest take away for me was learning to use that haptic motor driver, which allowed me to select a vibration effect which would create a smooth vibe instead of the rigid binary vibe. There are ways to produce similar results without the new driver we tested, but it’s not as fluid.

screen-shot-2019-02-27-at-8-02-13-pm

Creating the wearable
I pulled out one of the many old kid’s socks that have lost their match long ago. I cut the sock in half planning to use the top part for the wristband. The lower part of the sock I cut into strips and sewed them together to find my fingers. When I started this project, I wanted to create a stretch sensor that would run from the finger bands to the wristband. When the hand was clenched into a fist, the circuit would complete and set off the vibration motor. While I was playing with this idea, I thought it might be difficult to get a correct length that would stretch far enough without being too tight.

screen-shot-2019-02-27-at-8-03-34-pm

Exploring gestures
Once I had that revelation, I started playing around with the pieces of the sock and testing different gesture that I thought kids might do or understand. For the context of pressing a button, I settled on pushing two fingers together because it seemed to be the most natural. I also played with the positioning of the vibration sensor.

screen-shot-2019-02-27-at-8-05-55-pm

Connection sketch and execution
Once I had the idea, I drew out the circuit. I wanted to make sure it made sense before I started sowing to make sure I didn’t waste materials. After I wrapped my head around that, I started laying in the conductive thread. I used a flat stitch to create one side of the ‘button’ on the top finger band — This will be the part that is pressed up to the thumb band later. Then I brought the thread down to the lower finger band and stitched it to the front of the band and connected that side of the thread to the positive part of the LED and tied it off. Making sure they didn’t join, I stitched the negative side of the LED and sewed that line away and down into the wristband. Using that same line, I stitched the black wire on the vibration motor to the wristband (still using conductive thread). I did the whole process over on the other side, except I ran the fingertip band to the negative side of the LED and ran another line from the positive to the wristband below. Using that same line, I tired it into on front of my felt battery pack and tired that end off. Using a new line, I tired that to the other side of the battery pack and completed the line at the red wire of the vibration motor.

screen-shot-2019-02-27-at-8-07-25-pm

Trouble Shooting
Guess what, it didn’t work. Even though I was confident, it would work! I checked all the lines, and it looks correct according to my sketch. I tried flipping the battery so the negative/positive current would run the other way… nothing. I thought, maybe the exposed conductive tread on my skin was grounding the circuit, so I took it off and tested it… still nothing.

screen-shot-2019-02-27-at-8-09-15-pm

Plan B
Time for plan b, what if I can get this to work with a third LED? Maybe it’s the tread connection on the vibration motor which is loose when I was examining the prototype. I snipped out the motor and stitched in another LED again testing it without my hand inside in case that effected the outcome. Still, it didn’t work. I tried connecting the open wires to see if there was a point where the current would flow through. I touched the thread at the top finger bands, and that didn’t light anything up. Then I rubbed together the thread between the wristband and the lower finger bands and the bottom LED turned on — but only that one. I’m still puzzled… if you have any ideas as to why this didn’t work — Please share 🙂

Next steps
I wanted this to interact with an image on a screen in P5. The gear would provide instant feedback through haptic sensory and visual stimulation. I think kids would love the sequin LEDs all over their ‘glove’ with a vibration. It would be ideal to have this gear control a fun game on screen, and the interaction would be as simple as shooting at a target or jumping over obstacles when the ‘button’ is pressed.

Reminder String with Bracelet

Experiment by: April De Zen

screen-shot-2019-02-14-at-10-27-05-am

Idea
The purpose of this assignment is to create an ambient reminder, taking that into consideration I thought it would be neat to update the old string on your finger trick for reminders. The new twist would be having the string change colours on your finger to remind you. Since this will need some battery power, I added a bracelet to the design so the ring wouldn’t become bulky.

Materials & Tools
Thermochromic pigment
1 roll of conductive thread
1 roll of resistive thread
1 roll of regular thread
1 roll of yarn
1 piece of sheet felt
3x 3V round battery
Sewing needles
2 Alligator clips
Scissors
DIY French knitting kit

Workshop and experiments
In class we experimented with Thermochromic pigments. We dyed fabrics and threads and blended the pigments with white paint as well. We tested how resistive thread can be used to send a low current through the pigment in the fabrics to change the colour.

screen-shot-2019-02-14-at-10-34-55-am
Creating the wearable
After formulating my idea, I started making it. First using a DIY French knitting tool (link to instructions) I created the base of the bracelet and left extra yarn on each side to create a ring. My 1st bead was to use the resistive thread and use my friendship bracelet making skills to weave the thread into the yarn (link to friendship bracelet instructions). Once the thread was incorporated into the yarn I could start testing to see if the colour would change to white and how much voltage I would need — at least that was the plan.

screen-shot-2019-02-14-at-10-40-06-am screen-shot-2019-02-14-at-10-36-06-am

1st Failed Attempt
Turns out, the pigment on the thread brushed off if you handle it to much so as I was weaving the tread the pigments change right off and there was no colour left on the thread. I still ran a line of conductive thread for the resistive thread through the bracelet because I was convinced I would find a way to fix the colour issue.

screen-shot-2019-02-14-at-10-45-26-am

2nd Failed Attempt
Disclaimer: DO NOT DO THIS.
I had a thought that I would try painting the pigment on and for some reason I thought I would use the conductive thread instead of the resistive thread. I thought it would work better…
I wove the conductive thread to the yarn and painted the pigment on. When I tried to test it, it sparked immediately and blew out my batteries. Bad idea.

screen-shot-2019-02-14-at-10-47-30-am

3rd Failed Attempt
Back at the studio, I got my hands on some more pigment mixed in water do I could try dying the 1st attempt to bring the colour back. I thought this worked, it was bright blue and looking amazing. When I tested it, the thread turned white just like it should but when it was disconnected it didn’t go back to blue. It stayed white. Another fail.

screen-shot-2019-02-14-at-10-49-11-am

4th Attempt (not ideal)
Out of complete desperation, I another took a test piece of yarn and 5 strands of resistive thread and quickly knotted the yarn with the thread. Then I painted it to see if this would work. Maybe I could still save my original prototype by painting instead of dying the thread. Upon testing, it looked like it would work so I have it a go.

screen-shot-2019-02-14-at-10-52-14-am

Next Steps
I don’t know if I even want to take this any further. It is a cool idea but the truth is, changing from light pink to white has no impact and is barely noticeable. The out come really doesn’t warrant this much work, in this case. Not sure if it’s just a bad batch of pigments or if I missed an important step to setting the colour but I really didn’t enjoy this medium.

Dog Activity Sensor

Experiment by: April De Zen

screen-shot-2019-02-06-at-7-39-09-pm

This experiment I wanted to create a fabric sensor that could track a ‘swaying’ motion. I really like the idea of creating electronic sensors with house hold items and I decided to continue in that fashion. I will be creating two strands of beads with a metal bead at the end of each strand. When the metal beads touch it will complete the circuit and turn on the LED. When the sensor is completed, I want to create a dog collar that will track the motion of a beloved pet dog. After we have a LED lighting successfully, the next stage will be to hook the sensor up to an Arduino and track the collars swaying motion through the serial monitor.

Step 1: Materials & Tools

  • 1 roll of conductive thread
  • 1 roll of regular thread
  • 1 roll of wire
  • 1 piece of sheet felt
  • 1 3V round battery
  • 1 adafruit sequins LEDs
  • Sewing needles
  • 3 Alligator clips
  • A bunch of plastic beads
  • 1K resister
  • Scissors
  • Needle nose pliers

Step 2: Beading
Luckily, I didn’t need to buy anything for this experiment since I had a bag of plastic beads from some kids crafts and the rest of the materials I already had in my kit. I sorted out a nice variety of beads I wanted to use and set them aside. I had about 10 beads for each strand. I don’t have any medal beads so I decided to make some by wrapping wire around a pair of needle nose pliers until I started to see a nice looking ball. Then I make a loop with the end of the wire and tucked the end of the wire into the ball to make a loop. I’ll use the loop to connect the wire bead to the conductive thread later.

screen-shot-2019-02-06-at-7-43-30-pm

From there, I added a piece of conductive thread to the metal bead making sure to double up the thread so it will hold the current of electricity better. Then I started started stringing the beads on to the conductive thread and stitched the excess thread into a piece of felt sheet. One strand was stitched to the right and the other was stitched to the left. I added and LED light and stitched a connection from the positive to the right side and the negative to the left side.

screen-shot-2019-02-06-at-7-45-43-pm

Step 3: Testing circuit
When I plugged in the LED came on immediately so I knew I did something wrong. The LED shouldn’t go on until the metal beads touched and complete the circuit but it looks like the circuit was already completed and when the 2 beads touched it would short out the LED and drain the battery. I had to go back to the drawing board and see where I went wrong.

screen-shot-2019-02-06-at-7-47-08-pm

Step 4: Fixing the circuit
After drawing out the circuit, I can see where I went wrong. Connecting the LEDs to each side completed the circuit so I needed to keep the LED on one side. Since I would be hooking this up to an Arduino eventually I would also need a third wire (or thread) to connect to that. When I saw the circuit on paper it made more sense and I was able to map it out better. Following the drawing, I made a new circuit and this time — it worked!

screen-shot-2019-02-06-at-7-50-00-pm

Step 5: Testing with Arduino
I wanted to track the motion of the dog so when it was moving the beads should touch sending a signal to the Arduino. To do this, I used some sample code provided by Arduino. I used the ‘Digital Input Pull Up’ to test it out. If it worked properly this code will return a ‘1’ when the beads are touching and a ‘0’ when they are touching. According to the code I would need to use pin number 2 to track the signal. I connected that pin to the thread that came out the top of the sensor using alligator clips. I connected the ground to ground and made sure to add a 1K resister to the 3.3 volt just incase. I’m actually not sure this needed it but it worked.

screen-shot-2019-02-06-at-7-54-43-pm

Step 6: Having fun with Dog
The next step was to see if this sensor would even work with the motions of a pet. For the sake of science, I put it on my dog. I disconnected the sensor for my Arduino and taped off the third line to convert this sensor back to just battery operated. I used some leftover wire to twist around the conductive thread on both sides. After that I connected the wire to each side of the battery pack and tucked it behind the collar. The sensor worked perfectly so I attempted to put it on the dog.

screen-shot-2019-02-06-at-8-02-25-pm

This is Sparkles Joey Joey Superhero. Not a typo, my kids named her and they must feel very strongly about the name ‘Joey’. I had to bribe her with many treats but I got her to wear the collar without trying to chew it off. She ran around with it on and I noticed that the beads didn’t always touch when she was moving but it did light up many times while she wondered. I think another iteration of this should have more strands of beads and maybe a larger surface for the circuit to connect (longer metal bead?) I also wondered what would happen if she fell asleep with the beads touching. That would technically register as movement if the Arduino was hooked up to it.

screen-shot-2019-02-06-at-8-05-32-pm

Pressure Sensor Boots

Experiment by: April De Zen and Veda Adnani

This week was an intro to pressure and stretch sensors. There were 2 techniques that we covered in class, the first was clipping fabric and fibres with alligator clips and extending the textile to create resistance. The second technique was to create a ‘sandwich’ textile tool to hold resistant materials that will allow more current to flow once pressure is added.

After class, Veda and I thought it would be cool to create sensors that go into a persons shoe. Here are the steps we went through to build this prototype.

Step 1: Understanding the Materials
(Tests with Multimeter and Arduino)

screen-shot-2019-01-30-at-10-43-05-pm

In class, we all received a bag of goodies, many pieces of fabric and fibre to show the resistance of a textile. So the testing began, we tested everything. We tested stretch fabrics, a grey cotton ball thingy and more black materials. Each material created resistance in different fashions so we began chatting about different opportunities that these affordances could be leveraged. Some materials offered quite a bit of resistance and others not so much, it was fascinating exploring the materials in this fashion.

Step 2: Sensor idea

1

Next came ideation, what kind of sensor should we create. Light up shoes are fun for all ages, and we wanted to try making a pair ourselves. We chatted through the placement of the sensor, should it be under the heel or under the toes? The interaction for a sensor under your toes would be different than a sensor under your heel. We step heel/toe, and quite a bit more of pressure is pushed through the heel. Since the sensor was going into a shoe, it made the most sense to use a pressure sensor rather than the stretch sensor. Off we went…

Step 3: Assembly

screen-shot-2019-01-30-at-11-05-09-pm

There is a process of making a pressure sensor, we need two pieces of non-conductive fabric that gives a bit of a squish feeling when pressed. Next, we needed two parts for conductive fabric, in this case, it was fabric tape. Then we need a material to go right in the middle which will act as a resistor, we used the velostat which was given to us for testing. Once all the pieces were cut out, it was time to sow it all together. We stitched the conductive fabric tape in place (since it was shifting around) and then some conductive thread was sown into the conductive fabric to extend the reach since this sensor is going into a shoe. Then we quickly sandwiched it all together and stitched up the sides making sure the velostat covered the conductive fabric completely. We didn’t want to current to skip over the resister we just created.

screen-shot-2019-01-30-at-11-07-51-pm

Step 4: Testing in Shoe

screen-shot-2019-01-30-at-11-11-09-pm

The sensors were now ready for testing. We quickly tested it outside of the shoe to make sure it would work. Since the main idea was to rThe sensors were now ready for testing. We quickly tested it outside of the shoe to make sure it would work. Since the main idea was to re-create a light up shoe, we went with using the e-textile tool with an LED for testing. It was fun testing this sensor out, the LED lite up as we stepped on the sensor. Once it was placed into the boot, we pulled the conductive thread through a few holes in the boot so we could connect the sensor again to the e-textile tool while we took turns walking.

Reflection and Learnings:
This was an enjoyable exercise. Now that this exercise is completed it easy to see the simplicity in what we created but it is also phenomenal that current can be altered through fabrics with a few simple tricks and understanding what each material can do for you. Besides taking Halloween costumes to a whole new level, we are both very excited to use sensors like this in upcoming projects.

 

E-Textiles: Embedding a circuit into woven yarn and felt

Experiment by: April De Zen

screen-shot-2019-01-23-at-10-18-23-pm

For the purpose of this experiment, I created two conductive controllers using yarn, felt and some conductive textiles. The prototype created is a battery operated bracelet with a felt button which turns the lights on. This is my first try with e-textiles so I wasn’t really sure what to expect. I noticed through the process that without resistors controlling the flow of electricity it was very easy to burn out a battery which made the process more tricky then expected. Here was my journey creating and testing an e-textile.

Step 1: Materials & Tools

  • 1 roll of yarn
  • A piece of scrap cardboard
  • 1 roll of conductive thread
  • 1 roll of regular thread
  • Conductive felt
  • Regular felt
  • 1 piece of sheet felt
  • 1 3V round battery
  • 5 adafruit sequins LEDs
  • Sewing needles
  • 1 clean yellow sponge
  • 2 finger protectors
  • 1 felting needle

Step 2: Create Cardboard Loom

screen-shot-2019-01-23-at-10-34-33-pm

With the help of a tutorial from the weaving loom, I created a cardboard loom very quickly. I’ve never woven yarn before so I’m not sure how this holds up to a regular loom but I was really impressed how easy it was to make and how effective it was. It didn’t take mush time at all for create the loom and begin weaving. The pattern of a weave is very simple as well, under and over. The only issue I ran into was keeping the shape of the band. After a while I noticed that it was tapering in as I was weaving so I tried to loosen the yarn.

Step 3: Add conductive thread into weave

screen-shot-2019-01-23-at-10-37-02-pm

There needed to be two points of conductivity to flow electricity through the woven bracelet. Once I have about 3 centimetres woven I added the thread. I left some space without conductive thread because I will be adding a felt ball/button later and I don’t want it sitting directly on top the circuit. I pulled the conductive thread all the way to the other end of the loom and tucked it in at both ends to keep it still. Once that was complete, I continued the weave until I reached the desired length.

Step 4: Create felted ball with conductive felt for button

screen-shot-2019-01-23-at-10-42-38-pm

Creating the felted ball was an easy process but it is also very easy to poke yourself with the extremely sharp needle. Even with the finger protectors, I poked myself a few times. Go slow! Trust me on this. Since I was creating a button, I needed the conductive felt to be inside the ball so it would not trigger the bracelet until pressed. I started with laying the felt flat on the sponge (with the conductive piece on top) and started tapping the needle through. As I went I folded over the felt to create a ball and continued tapping the needle through. I made sure all the conductive felt stayed inside the ball while I was doing this.

screen-shot-2019-01-23-at-10-43-46-pm

Step 5: Add LEDs

screen-shot-2019-01-23-at-10-47-59-pm

This was my first time using these kinds of LEDs so I quickly tested them all to make sure they worked before I sowed them in. There was already two conductive threads running though the weave but I was worried it was not enough thread to seal the circuit. When it came to sowing in the LEDs, I kept a running line of thread through the positive line of lights and another running line through the negative line, making sure the 2 lines never touched. In the middle image you can see that I tucked the running line of conductive thread into the weave so it was hidden.

Step 6: Testing

screen-shot-2019-01-23-at-10-50-28-pm

When it came to testing I was successful with the felted ball but not so much with the LEDs that were sown into the woven bracelet. Out of five LEDs, I was only able to get one of them to light. It was confusing why only one worked so I decided to try some trouble-shooting to see where I went wrong.

Step 7: Trouble-shooting LEDs

screen-shot-2019-01-23-at-10-53-50-pm

Using the e-textile testing tool I created, I check the connections through out the bracelet. When placing the tested lightly on top of the bracelet it light up right away. Since I had already tested all the LEDs before the sowing, I was really confused as to why the others weren’t lighting. I loosened some parts of the weave to see what was happening with the connection points. It all seemed fine. I tried adding in some more conductive thread to strengthen the connection to the LED, no luck.

Step 8: Finishing

screen-shot-2019-01-23-at-10-56-50-pm

Even though I wasn’t able to see figure out why only one LED worked I still wanted to finish up the prototype to see if it all worked together. I tucked all the loose yarn from the weaving process into the woven bracelet and trimmed all the excess. It was starting to look like a finished piece. I added a felt battery pack to the back of the bracelet and snapped it in place. Again, only one LED lit. Then I added in the felt ball to see of it would trigger the light. I inserted the positive thread into one side of the felt ball and the negative thread into the other side making sure they didn’t touch each other. Unfortunately, it sucked all the power from my battery almost immediately. After replacing the battery I tried again but I noticed that when the felt ball was connected it didn’t turn the light off. The light stayed on and then I pressed the felt ball it would turn the light off. This was the opposite effect that I wanted and it would drain my battery very quickly.

Resources:
Creating a cardboard loom:
https://www.theweavingloom.com/how-to-make-a-cardboard-loom/

Felting for Beginners:
https://www.youtube.com/watch?v=fU6tihDWHhQ

Plus class tutorials and instruction