I created a simple game using machine learning that I call Frog Pond. The goal of the game is quite simple, you are a fly that must dodge frogs to stay alive. Using my laptop's camera and a KNN classifier script by Dan Schiffman, I trained the program to distinguish between multiple inputs. How this works is I would show a thumbs up and hit the "U" key to teach the program what up is, then I would do a thumbs down and hit the "D" key to teach it what down is. Depending on what the program thinks you are showing it, the player will move either up or down.
Train with: U, D, M keys
One of the first sketches I created for this project was a controllable square that can be taught how to classify different things. At this point all I had was a square that had the ability to go left, right, up and down but I was excited because I was fairly confident that I could make a game out of it.
I installed this P5 collider library and took inspiration from one of the examples made by the creator. The sketch spawns rectangles of various sizes moving towards the right side of the screen, after a rectangle's x position is greater than the screen width, it is reset back to zero creating an infinite stampede of rectangles. In the sketch rectangles turn black after colliding with the player(ellipse), I removed this feature and implemented a score system that would reset upon colliding with another object. The last I needed to do was play test and add sprites
I had a lot of fun making this project and adding sprites to the game. I implemented a score and high score system to encourage players to dodge the frogs. I had to get creative to add animations to the player and enemies because I could not get gifs/videos functioning with collision. Despite this problem, I was able to create animations by changing between two images, the first image would be displayed if the score was an odd number and the second image would be displayed if it was even.
The player sprite changes to this image while colliding with a frog
Challenges and Future Implementation
The biggest challenge I came across was getting a decent training set that worked with every user. In my first training set I attempted to train my program to go up when the user raised their hands, down when they put them down and remain the same height when the user T-posed. Although the game worked for myself and some other classmates, it did not work as well if you were taller or if the player was standing closer/further than where I trained. I was also using P5's web editor which had a file size limit so I had to make sure that the training set data was no larger than 10mb.
Given more time I would create a playable version for mobile devices and consider recreating this game using pose-net. I believe using pose-net would make the game feel smoother as the program would not be guessing between multiple inputs but rather it would be mapping the in-game player to the height of the user's head.
Github files (I created the project in the p5 web editor so downloading the project through github does not work)