For this first sketch I wanted to explore ml5.js and COCO to play around with the concept of merging an analog object with its digital counterpart. I was partially successful.

Intended user experience: 1) When the user throws a physical tennis ball in front of their camera, a virtual ball appears and follows their physical tennis ball as they throw it up and down in front of the camera. When their physical and virtual ball (tracking together) reaches a certain height, a “floor” is drawn and 2) the virtual ball takes on a life of its own – i.e., it continues to draw to the screen, now moving independently from the physical tennis ball, and drops from its height onto the “floor,” bouncing until it comes to a stop.

Results: I was able to get COCO to recognize the analog object (physical tennis ball) and then draw a virtual ball to the screen at the position of the physical tennis ball, drawing a “floor” (rectangle) when the balls reached a certain height. I was unable to decouple the virtual ball from the tracked object and achieve the second part of the intended user experience.

Video of the working components (part 1 of the user experience) (YouTube) 

Code (on P5.js)