Author Archives: David Oppenheim

Sketch #5 – David Oppenheim

Unity + Arduino

Moving Unity blocks with Arduino

Moving Unity blocks with Arduino

For this sketch I wanted to start getting more comfortable with Unity and Arduino. Working from a blog post by Erika Agostinelli I created a simple Unity scene and was partially successful in getting serial data into the scene to control the movement of a block.

The following video shows the Arduino sketch with both buttons outputting correctly to pins 2 and 8:

Video (YouTube)

This video shows the block moving right (but failing to move left):

Video (YouTube)

A fair amount of debugging failed to solve the issue…and so I have a cube that moves right, but not left.

Code, including Arduino sketch and Unity project (GitHub)

*I decided to add a tower of blocks to see what it felt like to topple them with my simple controller; not sure if it was connected to this change, but I subsequently got an error from Unity saying the port was not open. I was unable to resolve this issue.

Sketch #3: Physi-digital Ball Drop, part2 — David Oppenheim

For this sketch I wanted to go back to my first sketch and add state machine logic to be able to decouple the virtual ball from the tracked object. I also wanted to add poseNet functionality in with the existing COCO model functionality to play with two different types of interactions in the same sketch: holding a physical ball and interacting just with your hands.

Video (YouTube)

Code (on p5.js)

Sketch #1: Physi-digital Ball Drop — David Oppenheim

For this first sketch I wanted to explore ml5.js and COCO to play around with the concept of merging an analog object with its digital counterpart. I was partially successful.

Intended user experience: 1) When the user throws a physical tennis ball in front of their camera, a virtual ball appears and follows their physical tennis ball as they throw it up and down in front of the camera. When their physical and virtual ball (tracking together) reaches a certain height, a “floor” is drawn and 2) the virtual ball takes on a life of its own – i.e., it continues to draw to the screen, now moving independently from the physical tennis ball, and drops from its height onto the “floor,” bouncing until it comes to a stop.

Results: I was able to get COCO to recognize the analog object (physical tennis ball) and then draw a virtual ball to the screen at the position of the physical tennis ball, drawing a “floor” (rectangle) when the balls reached a certain height. I was unable to decouple the virtual ball from the tracked object and achieve the second part of the intended user experience.

Video of the working components (part 1 of the user experience) (YouTube) 

Code (on P5.js)