Machine Teaching Documentation || Machine Teaching Documentation
Professor Adam Tindale
October 28, 2019
For this assignment, my plan was to create a model that could distinguish between different fingers being held up using a classification model. I ended up going through many iterations, eventually trying to create that same effect using a regression after not being able to use the classification to compare more than two poses. I ended up having trouble with the regression as well and ended up totally changing my idea. I ended up using a regression model to create a game of pong that you play by moving your face.
To accomplish this, I altered code from Daniel Shiffman. His YouTube channel, the coding train, proved to be extremely helpful when it came to understanding how regressions and save/load functions worked in ML5. My final version is built upon his Feature Extractor Regression file. In his video and code, he creates a slider that allows you to train your code to guess between two or more things using a number. You train, say, one pose with the slider set to 1 and another with the slider set to 0. Once trained, your code will give you a number between 1 and 0 based on which one it thinks you’re closest to.
What I did to make this code my own was what I chose to make this number effect. If you were to show an apple and an orange, you could say if the number is between 0 and 0.5 print apple, and between 0.5 and 1 print orange. What I did instead was made the slider number affect a platform’s x value and turned it into a pong game. You train the slider left to right so it can learn your position and press a key to make a ball appear allowing you to play pong with your face (or anything else you’d like).
As of right now, everything is working without issue. One feature I wanted to implement was save/load functionality, but I found that the ML5 library had issues loading any trained models when it came to regression, although it had no issue with classification models whatsoever. I learned that this was on their end and that there was nothing I could do about it, although I was upset that it couldn’t be included for the time being. The JSON file that is meant to be loaded has still been included in my GitHub project so that I have it in case that in the future I’m able to use it but, for now, they’re not actually affecting the build at all.
If I had more time, there would be a few things I would change. First of all, I would fade the live video feed once you’ve trained the model. That way, once your game starts, you’re not as distracted by the video while still maintaining what I think is so cool about this application of ML5. I would also put a bit more work into the scale of the webcam video, as depending on the display you’re playing on it can become quite stretched out. Ideally, I would code it to determine how large it can go without losing its aspect ration which I think would make it a lot cleaner. I would also most likely put some more work into the assets, figuring out how to use real ping pong assets such as a ping pong ball without it being so hard to see if you have a white background (which is why I’m currently using a simple ellipse instead). I believe that lowering the opacity of the video feed could help drastically with this.
I think that using facial tracking like this could have some interesting applications in other 3D games. Not necessarily as the main controller, but for smaller things such as looking around corners ever so slightly in a first-person shooter for example. It would feel extremely immersive without feeling like you’re depending on a finicky technology in order to play the game entirely, which is in my opinion why technology such as Xbox Kinect can be so frustrating. It feels like something that would be a cool option in settings as opposed to an analog movement
To summarize this assignment, I'm really happy with how everything worked out. I feel as if I did a good job of working within ML5's strengths and made a solid site that works really well at what it's meant to.
Code on Google Drive
Please note-Code must be run through a web server in order to maintain full functionality.