© 2019 Mikhail Mata. All rights reserved.

[Atelier I: Discovery 001] Machine Teaching Project – “Cockroach uses a Ball to Play Piano Because You told it to” by Mikhail Mata

Title: "Cockroach uses a Ball to Play Piano Because You told it to"
A musical instrument that a person controls with their face. In order to move the cockroach and change the pianos notes, the person must move left or right to change the notes, either by moving their head or walking laterally. To manipulate the volume either crouch down or stand up.

Github Project Link:

For this project, I have decided to explore the idea of using body parts and interactivity as means to control an instrument. First of all, I was really Inspired by Foot or Mat Pianos, tongue drums and theremins. Instruments that use a wide range of motion to control the notes.


So I initially started my project by searching around the internet for ideas and codes that would be the basis of my project. Eventually, I found something that would work great. I found what was essentially a theremin that you could control with your mouse. I figured out that this was great as it was close to the vision I had in mind. I expanded on this idea by streamlining the code overall and by replacing the mouse cursor with the nose key point from poseNet to control the X and Y values of the cursor on the canvas and change the notes. In addition, I also set the X value of the cursor to 0 so that it would only move laterally thus making it smoother to look at and more convenient for users to use. I also added a background image to hide the video and added other additional images for aesthetic purposes.

demoFor the most part, I'm pretty happy with how the project turned out. Asides from struggling with the aesthetic side of the project such as its layout, I achieved what I wanted to achieve and it works accordingly to how I want it. If I was given more time, I would definitely focus more on making it look more aesthetically pleasing and instead of having it mapped to the users face, I would have the pitch mapped to the right hand and the volume to the left hand, just so its more expressive for the audience. If the project was more polished it would be a good public installation that the any person could walk up to and interact with.

The HTML CODE:html-code




Demo Video:




Leave a Reply

Your email address will not be published.
Required fields are marked:*