Body as Controller / Making Web Games

Objective: Learn how one’s body or body parts can be used to interact with objects on the screen, where the body’s gestures becomes input or the controller.

Interacting with objects on a screen:

We hit a snag and discovered that there’s not much documentation on using openframworks with Kinect. And that the body tracking we wanted to do is possible with Kinect V2 with Microsoft SDK which is exclusive to Windows. We also don’t have access to the Kinect V2. An alternative Daniel Shiffman uses the Kinect V1’s depth tracking of a single pixel to “track a body part” in theory, however this method isn’t as intuitive or seamless as when using PoseNet. The problem with PoseNet is that it is difficult to differentiate unique poses as all the poses are stored in an array and you get back the latest pose…perhaps I could play with the interval between saving poses…but i think this may mess with the real-time tracking aspect.

Hand-position detection.

  1. Web-based using p5.js and PoseNet

I began with a tutorial on the which is a game library for p5.js. The good thing about this library is that you don’t have to do math. It handles the calculation of where to position your sprite based on it’s x,y position and the velocity you set for the sprite. So when you change the x,y position the library does the calculation for you.

prt1 . I worked on Allison Parrish‘s tutorial on how p5.js and did the following activities: Making a single sprite, sprites on the move, following the mouse, mouse events, multiple sprites, events on multiple sprites.

So far I really like how the library works and the functionality it provides. I was able to mimic gravity, causing rectangle sprites to fall down, bounce back up when they got to the ground, disappear when the mouse was waved over them, set a score when a sprite disappeared, and handle multiple sprites on the screen. I wasn’t too sure how to add my own function to the Sprite class. I may have to create a custom class that is of type Sprite i.e. extends Sprite of maybe I can create a custom class that has a Sprite as a variable.

From my explorations today, I’d like to look into how I can remove the spr.onMouseOver() and use a body part as the mouseover so that for example when you wave your hand over a brick it would disappear. I’d also like to continuously randomize the position of the bricks so that they aren’t static and are more challenging to swat.

i finished working on the second part of Allison Parrish’s tutorials. I did the following tutorials. Sprite groups, collisions – overlap(), collide(), group collisions, collision callbacks, images and animations, animations.

Things I learned:

  1. The overlap function and collide function determine the kind of interaction two sprites can have. Overlap() allows one sprite to move on top of the other, and collide() makes the second sprite solid blocking the other sprites path. How it works:

spr1.collide(spr2); // Sprite 1 will collide with sprite 2

spr1.collide(sp2); // Sprite 1 can move ontop sprite 2.

  1. The Group() class allows you to create multiple of similar sprites and save them together almost like an array. It uses add() and remove() functions to build the group.
  2. To create an animated sprite, you need three image variations of the sprite to be played to create the animated effect.
  3. To keep score, and display points you can use a global variable score and update it whenever a sprite is removed.
  4. collision callbacks allow you to call a function whenever two sprites overlap or collide, so If a player overlaps a coin you can have a getCoin() function and if a player runs into a wall you can have a getAHeadache() function. This function takes the player sprite and the other sprite as parameters.

Constraints: Because poseNet updates too quickly with the poses, I think it isn’t a good choice for a game where you need complete accuracy. I was thinking of making a snake game, but I can’t figure out how to use the body part to control the snake because the body part doesn’t move in a straight line but the snake needs to. I think I might come back to this idea but I’ll begin with a simple paddle game where the hand position controls a paddle. I will keep the y position constant and just have the x position change according to where the player is holding their hands so that they don’t have to keep their hands in a certain position and thus simplifies the interaction. 

Step 1. Creating the paddle.

I was having trouble getting the poseNet model to track my wrists. The shapes would jump all over the screen. Only the nose and eyes worked best. I think this may be due to the area the camera can see and also the light in the room. since my face is closer to cam, it it easier/clearer to track. I switched to using Google’s creatability library and had better results however, this meant that I couldn’t track both hands to create two paddles.

The paddle is created using a simple rectangle shape whose y value is constant i.e. 380 and whose x value is updated depending on the x value of the right wrist.


Step 2. Creating Walls

When the user’s hand goes off the screen the paddle follows off the screen too. To prevent this I created 4 wall sprites set to collide with the paddle so that it remains in the canvas. For this I created a group of sprites called walls and set the player to collide with any sprite from the wall group.



Step 3. Adding a bouncing ball

  • Set the ball to fall and when it collides with the paddle, stops falling and to test the interaction, I set the two to change colour upon colliding.

screen-shot-2019-07-06-at-7-47-52-pmthe ball sprite before collision with paddle


the ball sprite after collision with paddle

I was able to get the ball to bounce off the paddle and the walls post collision, however i wasn’t able to get the ball to speed up. Whenever I added the speed, it would bounce and go off the screen. see: bouncing ball sprite

Step 4. Adding complexity

I was able to speed up the ball a little bit by upping the velocity in the y direction. To add some complexity, I added a killer group that reduces the size of the paddle upon collision, killers are randomly generated every time a player successfully bounces the ball. I also added a coin group, that allows the player to score points if they collect falling coins.

Step 5. Controlling using the body

I switched out the mouseX for pX which is updated using the x position of the wrist, tracked using Google’s creatability library. The value is used to update the velocity of the player. I noticed that when using one’s hand as the controller, the paddle becomes a little jittery and the movement isn’t as fluid as when using mouseX. This could be because of the light in the room. Moving away from the webcam seems to improve things a bit. Also, running the webcam and the library at the same time seems to overwhelm the browser, and i get lags that didn’t show up when I was just tracking the mouse.

player.velocity.x = (pX – player.position.x)*0.1;


I ended up having a lot of trouble trying to run Creatability library with and ended up switching back to PoseNet on ml5.js. I also switched tracking to the nose instead of wrist, as this proved to be much smoother. Unfortunately the webcam is still flipped, so having the skeleton showing as the player plays is a little disorienting as movements are flipped, adding the drawing actions to the translation matrix that flips the video didn’t change anything.

link to code on github: here