October 4, 2020
Experiment One – Creation & Computation
K. Hartman, N. Puckett – OCAD University
Human Tetris is an attempt to combine a tactile full-body game with a modern adaptation of one of the earliest human-camera interfaces. Formatted as a five-level interactive game, the project progresses through a set of puzzles in which the player must contort themselves to stay within the bounds of a ‘tripwire’, displayed on-screen. The entire experience is intended to be navigated, played, and completed at a distance of around six feet; a custom user interface was designed to allow the user to control the game flow from this distance.
The project is inspired in part by the Brain Wall game show phenomenon and the pioneering user interface designed for the PlayStation Two’s EyeToy peripheral (see below). Engineered around the principles of discoverability and learnability, the project provides a minimum of instruction or tutorial to the user and lets them instead discover both the navigation and the play mechanism. Visual feedback such as incremental shape fills and text reminders ‘train’ the user on desired behaviour.
A key goal of the project was smooth, fluid navigation using the same paradigm as the game itself: the body should itself be the controller. The initial splash screen greets the user with a game title and a circle labelled ‘Begin’. By chance or by exploration, the user waves a hand over the button and sees immediate feedback: the button fills clockwise and progress circles appear akin to a clock face. This instantly conveys a time-based interaction and compels the user to hold or change position to impact the animation. This is the solitary navigation mechanic for users, which keeps the experience streamlined and approachable.
In the opening levels, the primary goal is situating the user in the activity area and providing them the basic game mechanics. Once the user has acquired the primary navigation mechanic (see Study One), the next aim is to convey how to successfully complete a level. The user is informed of their correct positioning by three key elements: a change in the tripwire stroke from yellow to green, a gradual sweeping fill of the tripwire area, and a ‘Hold it!’ banner that shakes to imply a strenuous core workout :).
Once the primary navigation and game mechanics are acquired, the game moves away from basic, symmetric tripwire shapes and into shapes requiring more significant contortion. Because up-down movements are reflected one-to-one via webcam (as opposed to left-right movement, see below), this interaction is the next logical challenge to the user. It is important that although the vertical shape of the body must be changed, the tripwire remains horizontally symmetrical so as not to overly disorient the user. This level in particular can require the player to make use of the space in front of and behind them; they must project their 3D body shape onto a 2D place.
The mirror image effect inherent to webcam-based interaction provides the opportunity to challenge and disorient the user. This asymmetric, jagged tripwire requires more sophisticated movement. Usually, this is the most challenging level yet for players as they must separately map their vertical movement and horizontal movement, which is inverted onscreen. In terms of establishing a smooth learning curve, this level was the most difficult to design. A balance must be sought between frustration and skill development. This may be the first level to present a genuine challenge or require several attempts on the part of the player.
As the final level, this tripwire tests the knowledge acquisition of the previous levels by requiring horizontal translation, tilting of the body, extension of the limbs, and an extended holding of a matching pose. By now you must assume that the game’s mechanics are internalized and familiar to the user, allowing the level design to be foregrounded. Completion of the level prompts a congratulatory splash screen before navigating the user to the home screen to permit them to play again. The game can be played on a loop indefinitely, with no further mouse/keyboard interaction required.
The concept for the game’s tripwires came from the Japanese game show phenomenon known as Brain Wall or nokabe, in which contestants are required to contort their bodies to pass through an advancing screen containing a specific body-sized cut-out (“Brain Wall”). The segment quickly became popular in Japan and went viral around the world (Fermoso). The format’s slapstick humour and the immediacy of the presented challenge helped versions of the show proliferate throughout more than 40 countries (Hollingworth).
Using nokabe to incorporate the goals of learnability, fun and seamlessness into my studies, I was encouraged by early demonstrations to friends and family that suggested the challenge was clear and the learning curve was shallow enough to be approachable. Over a number of iterations, I refined the difficulty curve of the experience until I was introducing one mechanic or degree of freedom per level.
The concept for the interaction ‘buttons’ was based on my experiences growing up with the EyeToy, an early augmented-reality experience built for Sony’s PlayStation Two that combined a small camera with a variety of games in which the player interacted with onscreen elements (Kim). I was interested in adapting the user experience for a world where, fifteen years on, machine learning and real-time pose detection are standard fare.
In EyeToy, menu elements were ‘active regions’ onscreen, demarcated with shapes or colours, that required a player’s body part to intersect with those regions for a short period of time (Zhou 99). Developers found that when adding a live video feed on screen, “people immediately grasped the control and interface [as] they were able to see themselves and receive immediate feedback from their on-screen actions” (Kim). I wanted to emulate that intuitive grasp of the interface so that gameplay was foregrounded rather than the user’s acquisition of interaction mechanics; employing the user’s hand as a cursor for onscreen buttons has been demonstrated as highly effective at reducing error rate and mean task time (Cheng 296).
Whereas the EyeToy’s human detection processing was based on pixel analysis of highly compressed individual frames (Kim), I was able to relatively easily incorporate live pose detection and tracking using the p5.js platform and ml5.js extension library.
The point-in-polygon math and circular animations ended up requiring significant development time. Of particular interest was efficiently solving the “pose inside tripwire” problem. Although it is a trivial calculation to the human eye, detecting whether a series of points all fall within a convex polygon required some intermediate mathematics. The even-odd rule (Hormann 131) allowed me to assert that a point was inside the tripwire by confirming that a line arbitrarily extended out from that point intersected with all tripwire line segments an odd number of times. This enabled the use of the relatively efficient line-line collision algorithm to confirm that the player remains fully inside a given shape (Hallberg).
Cheng, Kelvin, and Masahiro Takatsuka. “Initial Evaluation of a Bare-Hand Interaction Technique for Large Displays Using a Webcam.” Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems – EICS 09, 2009, doi:10.1145/1570433.1570487.
Fermoso, Jose. “Viral Video of the Year Nominee: Human Tetris.” Wired, Conde Nast, 20 Dec. 2007, www.wired.com/2007/12/viral-video-of/.
Hallberg, Chris. “LINE/LINE.” Collision Detection, crhallberg.com/CollisionDetection/Website/line-line.html.
Hollingworth, William. “Japanese Game Show Takes the World by Storm.” The Japan Times, 10 Mar. 2010, www.japantimes.co.jp/news/2010/03/10/national/japanese-game-show-takes-the-world-by-storm/.
Hormann, Kai, and Alexander Agathos. “The Point in Polygon Problem for Arbitrary Polygons.” Computational Geometry, vol. 20, no. 3, 2001, pp. 131–144., doi:10.1016/s0925-7721(01)00012-8.
Ishibashi’s Room Entertainer Edition. “Brain Wall – English Sub”. Online video clip. YouTube. Google, 19 Mar. 2020. Web. 24 Sep. 2020. https://www.youtube.com/watch?v=UY9axBEJy8s
Kim, Tom. “In-Depth: Eye To Eye – The History Of EyeToy.” Gamasutra, 6 Nov. 2008, www.gamasutra.com/view/news/111925/InDepth_Eye_To_Eye__The_History_Of_EyeToy.php.
Zhou, Hanning, et al. “Static Hand Posture Recognition Based on Okapi-Chamfer Matching.” Real-Time Vision for Human-Computer Interaction, 2005, p. 99., doi:10.1007/0-387-27890-7_6.