Machine Learning: PoseNet

By: April and Randa
Link to sketch code: https://editor.p5js.org/aprildezen/sketches/gkVoUOBeo

Strategy
To explore and understand how PoseNet works and how it can be altered and changed. We don’t have a lot of coding experience so we decided to take this slow and really understand the tutorial by Daniel Shiffman. Once we understood that we wanted to see what the limitations were and what we could build using this platform.

screen-shot-2019-03-09-at-2-56-00-pm

Documentation
Following the tutorial, we went step by step to create the canvas and set up the video capture. Before moving on we played with a few of the filters provided in the P5 library.

screen-shot-2019-03-09-at-2-17-46-pm

Once that was set up we moved on to setting up PoseNet. By using TensorFlow.js PoseNet model, the computer pulls up poses that match ours in real time, and displays skeleton-like graphics of our body. PoseNet can be used to estimate either a single pose or multiple poses, but just for simplicity we experimented with the single pose that detects only one person in a video.

We copied into the p5.js editor’s index.html file the below url for the ml5.js library: <scriptsrc=https://unpkg.com/ml5@0.1.3/dist/ml5.min.js”type=“text/javascript></script>

Importing this library allows the program to detect and trace keypoints positions in the face and body using the webcam. Then one can create interactive art or graphics that respond to body and face movement.

All keypoints are indexed by part id. The parts and their ids are:
screen-shot-2019-03-10-at-7-37-41-pm

When it detects a person, it returns a pose with a confidence score and an array of keypoints indexed by part id, each with a score and position.

screen-shot-2019-03-10-at-7-37-50-pm

This information allowed us to identify the positions of each pose keypoints, and also the confidence score of detecting them. After retrieving this information we played around with the code and tried out assigning shapes to different positions and keypoints. We learned also that to make shapes scalable we need to input their size to be a variable defined by the distance between the nose and an eye.

screen-shot-2019-03-09-at-2-25-30-pm

This is another example we explored that creates wobbly eyes.

screen-shot-2019-03-09-at-2-28-35-pm

Insights
We were both really blow away with how accurate PoseNet’s location detection was. We both played with it alone on our computers in class and it was eire how other it was right.

However, There is a lower accuracy detecting keypoints of the body with multiple people then there is with just one person. We played around with a few demos and noticed the lag time increase when we were both in view. In the video above, the clown nose was juggling between both our noses, trying to detect the more confident output. It actually was kinda a fun effect, we felt it felt like playing digital ping pong with our noses.

Now that we played with the demos, time to see what we can do with it.

WELCOME TO OUR ALIEN ABDUCTION.

screen-shot-2019-03-09-at-2-33-05-pm

Unfortunately, we forgot to hit record on this part because we had a lot of fun trying to figure out how target other points in the PoseNet library and add silly alien anteni and understand how the example code for the googly eyes worked.

Here is the breakdown of what we went through to become extraterrestrials.

Step 1: Locate where the ears are
Using the example code from the hour of code tutorial and the index of part id’s, we could easily locate the ear location – Left Ear 3 and Right Ear 4. As a placeholder we drew two ellipses over the ears to make sure it was working.

Step 2: Create anteni above our ear location
Now that we had red ellipses floating on top our ears, we needed to figure out a way to move them up above the ears. Since we just wanted to move them up, we only needed to change the Y axis. In the string that indicates the placement of the ellipses over the location of the ear, we made one slight change. We multiplied the the ‘ear1Y’ variable by .5 and that gave us enough height.

ear(ear1X, ear1Y*0.5);
ear(ear2X, ear2Y*0.5);

To create the anteni we simply change the x and y height so that the ellipse would we long and skinny and change the red colour to a RGB value that will give us bright green.

function ear(x, y, size, n) {
fill(164, 244, 66);
noStroke();
ellipse(x, y, 5, 100);
}

Step 3: Bring in the googly eyes
The googly eyes was part of the demo code that was included in the hour of code tutorial we watched on PoseNet. All we changed for this was the frame count to 2 so the eyes would spin faster and we changed the eye colour to green.

function eye(x, y, size, n) {
let angle = frameCount * 2;
fill(255);
noStroke();
ellipse(x, y, size, size);

fill(164, 244, 66);
noStroke();
ellipse(x+cos(angle*n)*size/5, y+sin(angle*n)*size/5, size/2, size/2);
}

Step 4: Add a filter to the video capture
The last thing we did to achieve this strange alien look was to add a filter to the draw function.

filter (INVERT);

Final Obduction #OOTD

screen-shot-2019-03-10-at-7-16-48-pm

Information sources
https://p5js.org/learn/color.html
https://p5js.org/examples/dom-video-canvas.html
https://p5js.org/reference/#/p5/filter

Reflection
Overall, we had a lot of fun playing with this. There is so much more we can do with it and though this learning experience we noticed how achievable it was for us to use it. Neither of us know much about javascript but we were able to figure out how it worked and how to start making changes. We think it would be interesting to continue exploring this tool.