Experiment 3: Material as Sensors
Naruto Glove Game Controller
Madelaine Fischer-Bernhut, 3161996
Salisa Jatuweerapong, 3161327
Brian Nguyen, 3160984
Atelier 1: Discovery, DIGF-2004-001
15/11/2018
Concept
A Naruto Primer:
From Wikipedia: “Naruto (ナルト) is a Japanese manga series written and illustrated by Masashi Kishimoto. It tells the story of Naruto Uzumaki, an adolescent ninja who searches for recognition from his peers and the village and also dreams of becoming the Hokage, the leader of his village.”
The Naruto franchise is one of the most internationally well-received and popular of Japanese IPs– ranking as the third best-selling manga series in history. This pervasiveness of the IP gives us a solid launching point for our project; the majority of people should at least have heard of it, while for others it invokes deep nostalgia and childhood memories of mimicking hand jutsus in hopes of creating fireballs and wearing dorky headbands and cosplays for play-acting.
The main power system of Naruto revolves around chakra, a “life energy” that all ninjas can tap into and channel into a variety of attacks and powers (jutsus) by using hand seals. Hand seals (Hare, pictured on the left) are specific hand gestures, and different combinations of them create different types of attacks. There are 12 basic hand seals (based off the zodiacs).
Since its initial birth, there have been over 50 Naruto games released and the majority of them are fighting games that rely on jutsus among physical attacks. However, for the most part they are all played using generic consoles, ie button presses. The game we are using: Super Smash Flash 2 (Pictured on the right), is a fan-made flash game made in homage of the Super Smash Bros series. In this game, the creator decided to add many popular characters from anime and gaming that are not typically found in the original series. For example, Goku from the Dragon Ball series and Naruto from the Naruto franchise, which we are using for our project.
Our concept is to bridge the gap between the physical world and the digital world through motion gesture controls that put you in the shoes of the playable character. Instead of pressing a button to create a hand seal inside the game and form the jutsu, the player would be able to form the hand seal in the physical world and have it reflected by the character in the game world; instead of pressing arrow keys to move your character around, the player would be able to step in a direction and have the character on screen copy their movements. In the spirit of childhood nostalgia and avid fans, we are turning people into ninjas.
Group Member Roles
Madelaine: Creating the positive glove, Leaf Symbols, sensor placement plans, and forearm guards creation.
Salisa: Initial conception, creating the grounded glove, sensor placement plans, and movement mat design and creation.
Brian: Programming the Arduino Micro and finding a workable game.
All: Conceptual planning and circuit creation, documentation.
Initial Planning
Placement of sensors
Image 1: Initial sensor/switch placement colour coded
Image 2: Updated sensor/switch placement colour coded
Image 3: Sensor/switch designing (shape, type of stitch, positioning, etc)
When we were planning the the placement of the switches we had to make sure that we would not accidentally trigger the wrong switch when making a hand sign. The first step was to only choose hand signs that did not have that many of the same points touching. We decided on Boar, Dog, Ram, and Hare because their signs almost all made contact with at least one different point. At first, we were thinking of creating a switch by putting the ground and positive sides of the circuit on the same hand. The other hand would be unattached to any wires and bridge the circuit (acting as an activator only). We decided against this idea because we realized that it would make sense to have one of the gloves be the ground to limit the clutter of too many wires. Because of this decision, the left hand contained the triggering switches connected to each gesture.
Hare and Ram are the only gestures that required an override of another sensor, the palm switch. Ram we knew from the beginning had a point overlap, but for Hare, it came to us when we were testing the performance of the gloves. Ram’s unique point of contact was the middle and index finger and it shared the palm point with the boar. Hare also shared a palm point with boar depending on the angle the player used in making the gesture, so we had to make sure its unique point, crease of pinky and ring finger, overrode the palm point.
Materials
Hardware: Arduino Micro. It has the ATmega32U4 chip which allows it to act as a keyboard when connected to a computer through USB. The full support of being able to program keyboard keys to the micro was extremely helpful in writing the code. Before this, we thought of alternative ways to simulate keyboard inputs such as a Python script or a Chrome extension listening on the serial port. As well, Donato suggested using a Makey Makey.
Gloves: A black knitted glove, black thread, silver (non conductive) thread, conductive paint, conductive fabric, and conductive thread.
Movement Mat: Velostat pressure sensitive material, aluminum foil, neoprene, and fabric glue.
Forearm Guards: Black poster paper, neoprene, electrical tape, black thread, and fabric glue.
Implementation
Gloves
While we originally debated sewing our own gloves (this way we could fully match the design and create the entire ground glove using conductive material), we ended up grabbing 99 cent gloves from Chinatown because of the time crunch. We chose knit gloves over leather ones for the price point and for the one-size-fits-all quality. These gloves stretched rather well and were able to fit grown men with larger hands.
The biggest issues we ran into were the shoddy construction of the gloves (we had to patch up at least 3 holes that just opened up) and the stretchiness of the knit, which made it impossible to iron on the conductive fabric as we originally planned. It was also difficult to sew evenly-placed stitches; the craftsmanship was a steep learning curve.
Outside
Image 1: Back of gloves Image 2: Front of gloves
Inside
Image 1 (from left to right): Conductive fabric “wire” connections for the pinky and index finger crease, the middle finger, and the Backplate.
Image 2 (from left to right): Conductive fabric “wire” connection for the Leaf Symbol and the ground.
*the paint cracked by the end of the day, unfortunately, leaving just half a leaf
The sensors/switches were made with either conductive thread to be less prominent as possible or conductive fabric to create more intricate designs. Each sensor/switches positive side was connected to the alligator clips, going to the Arduino Micro, using a thin strip of fabric inside the glove. The reasons for this was:
- We wanted the gloves to remain as close to the original as possible. This did not include silver thread crisscrossing the black glove.
- If we used thread, the stitches on the outside of the glove may accidentally trigger the wrong hand gesture if the wrong sensor is touched by the ground.
- We did not want to use wires as they would be too rigid and would make putting on the gloves difficult.
We decided to stick with the black gloves and Leaf Symbol design because they are so iconic in the Naruto Universe. The black gloves with the metal plate on the back are a signature accessory for Naruto’s ninja mentor, Kakashi. The Leaf Symbol is the symbol of Naruto’s home village in the franchise. Both Kakashi’s gloves and the leaf symbol are extremely popular in Naruto themed merchandise and cosplay, so Kakashi’s gloves are usually designed with the Leaf Symbol (the plates in the canon universe are plain) and pretty much accepted as fan-canon.
The Leaf Symbol for the palm switch was a stylistic choice to keep with the theme. Many of our initial ideas were basic palm padding designs. We ended up scrapping the idea because we felt that knitted gloves usually do not have pads and that the silver of the fabric would be too different from the original black glove.
The biggest issue we had creating the sensors was with the conductive thread being covered with the thick fabric of the knitted glove. Even after adding more thread to the spots with the sensors, some of the hand gestures had issues triggering (the Hare and the Ram)– mostly in part, due to larger hand compositions. Sewing the conductible fabric designs were also difficult. The material is extremely stretchy so it was easy to make a placement mistake that would look normal when unworn and look extremely off when worn; or vice versa. The conductible threads tendency to twist and knot were also banes of our existence.
On a stylistic note, the conductive stitching on the seams were meant to create “invisible” sensors; however, the stitches sunk too far into the fabric if they were not bunched up, and were not so “invisible” if they were. We stuck with it on the ground glove as it was needed to create the circuit, but avoided it on the right glove as the look was not as polished as we imagined. If we were to use a different material for the glove (and get really good at sewing), stitching matching seams on both gloves would effectively hide the sensors in plain sight. In addition, using black conductive thread.
Hand Signals/Gestures:
Key: I – Move: Defense Barrier/Shield
Switch Sensor Location: Backplate of the right glove (positive), Leaf Symbol on the palm of the left glove (ground)
Key: O – Move: Special
Switch Sensor Location: Conductible thread seam in the crease between the pinky and ring finger on the right glove (positive) and between the thumb and pointer finger on the left (ground)
Key: P – Move: Basic Attack/Smash Attack
Switch Sensor Location: Leaf Symbol on the palm of left (negative) and right gloves (positive), conductible thread sensor around the middle finger on the right (positive and left gloves (ground)
***For the longest time we thought this gesture was the Tiger sign. So please do not mind the slightly off finger positioning in documentation photos.
Key: U – Move: Grab
Switch Sensor Location: Leaf Symbol on the palm of left (ground) and right gloves (positive)
One thing we debated was making the hand seals work even in mirror position (mostly for Dog) as many players mixed up their left and right hands. We ultimately decided not to as it went against the spirit of the ninja way. As a hypothetical commercial decision, this controller is marketed towards fans, so it should not be an issue– in fact, making them work in reverse would likely be a complaint from hardcore fans.
Testing
Every time we finished an element of a switch (positive or ground), we tested to make sure was able to run through the circuit by using an LED and a battery. The placement of the Leaf Symbols, the Backplate, and the sensors/switches were roughly figured out using tape on our hands, then on the glove (as well as a fabric pen later on).
Movement Foot Control Mat
Image 1: initial concepts for floor mat
Image 2: WIP of floor mat
For the concepts, we conducted research by simply walking around and trying to figure out how to create a natural body sensor for moving. This was a scrolling 2D game, so a 3D floor-mat like DDR did not make as much intuitive sense to us; should the up arrow be jump, or be walking forward on the screen? Our project was aiming to connect the player to the onscreen character as much as possible, so a similar movement was key. The other key design focus was ease of reach: we wanted the player to not have to search or reach far for the sensor, and ideally always be to press the sensor without moving their foot placement.
Based off the way the characters stood on the Smash screen (and other Tekken-style games), we came up with the layout pictured on the lined paper: a ready-position half-crouch, with one foot in front of the other. The drawback with this was that it was not intuitive at first glance (though it would still be nice to playtest; it has potential, especially since we can use code to make it work), and most importantly, the sprites in this game could flip around on screen. So even when the sprite would be running left on the screen, the player would be pressing back with your left foot, and the focus became more on the left-right than the back-front.
Therefore, that idea was scrapped for a simpler left-right 2-sensor design. On top are some Naruto-themed designs, including ones based off significant clan symbols, weapon designs, and for one, a symbolically geographical representation of Naruto’s clan maps (there was a lot of overthinking).
Our biggest grievance with the 2-sensor designs was how… untethered, they felt. Both in position in relation to the other (they could easily be set-up differently or kicked apart) as well as in relation to the digital game and other gear. It raised the question of where the player would stand, at rest or otherwise: an space in between the two could work, but would have a significant delay in reaction time, meanwhile a space behind the sensors could make the player feel removed. With that in mind, and the map idea– we decided to make a single mat, with 2 sensors on either side. By marking a specific play area, we created a deeper sense of player immersion. Keeping with the colour scheme, red carpet fabric was glued to a sizeable neoprene mat. The tactile raise and feel of the carpet fabric helped players find the sensors with their feet. So the players would not have to lift their feet/ leave the play area, the sensors are located in the balls of the feet only; players can rest on their heels and rock forth on the balls of their feet to activate the sensor. (This movement needs playtesting and research for strain in continued use and should be adapted accordingly for safe play.)
The three comma-like shapes in the middle form the Sharingan, another nod to the Naruto franchise. Currently it does not do anything, but there is the possibility of 1) turning it into a jump sensor (up) 2) turning it into a duck sensor (down) 3) turning it into a special move where the player crouches down and presses their hand to it (it would likely be an analog sensor so any pressure works.) However, staying true to the franchise, there are seals and moves that require a ninja to press their hand onto the ground.
One critique of the floor mat design was that the player intuitively inferred that rocking forward would be moving forwards on the game, and rocking backwards would be moving backwards. This, while a cool mechanic, runs into the same issue as the initial half-crouch concept. The main issue here, however, is that the design does not clearly convey to the players to rock forwards to activate the sensor. This could possibly be changed by making the back heel an empty silhouette (we do want to mark where to stand regardless.).
On the day of the presentation, we were unable to get the movement mat to work due to a circuit issue. We knew something was wrong when we realized that the pressure values were way too low with little difference between rest and active states. Thankfully, right before the class presentations, we were able to rearrange the circuit and make it work properly. We made the threshold 500, so the movement would be easier to control.
Forearm Guards:
The forearm guards were our design fix for both organizing the wires to allow greater ease of player movement and set-up, as well as (somewhat) hiding the circuitry. We always knew that it would be best to get the wires out of the way by bundling them along the arm with some sort of band, and then upon further research we were able to design a themed cover that would hide the wires as well.
These guards are based off Kakashi’s Anbu uniform in the storyline (the gloves themselves are also based of Kakashi’s design). Many of the players mentioned how they felt cool and in character while wearing them, and that they helped with the immersion. The wires were organized, separated, and attached to the inside of the guard with electrical tape. Organizing the wires also made it easier to keep track of how the sensors were connected to the circuit.
In order to fasten the guard to the arm, two bands were created. Black poster paper was used to make the band permanently attached to the bottom of the guard near the wrist. The top band, on the other hand, was made from neoprene and was able to be placed anywhere to adjust to different sized arms. Both bands were fastened using Velcro dots.
A second prototype would likely have the forearm guards made of a mixture of foamcore and neoprene, and properly painted and textured.
*The neoprene strips were too short, so multiple strips had to be sewn together. Furthermore we ran out of neoprene, or we could’ve used them for all of the straps.
Circuit
*We taped down the breadboard so it wouldn’t get knocked over during demos.
Code
Link to code: https://gist.github.com/notbrian/f1bdf661cceba52939de42252c553623
The code has a couple of different states it runs through during every loop.
- Checks the value of every pin, whether the circuit is closed or open.
- Loops through the state of these pins, and presses the keys associated with them
- Delays the program 10ms (Required for the game to register actions properly)
- Releases all the keys that were pressed
When designing the code, one of the most important elements we wanted to uphold is responsiveness. This element of controlling the game is especially important in fighting games like Smash where the milliseconds count when performing different moves which could result in behavior that the player doesn’t expect.
Originally the code had a delay associated with every key press and key release. This led to issues playing the game however. Firstly, the player couldn’t use more than one key at once which lead to unresponsiveness in gameplay. Secondly, the player couldn’t use “smash attacks” attacks in the game that are extra powerful but only triggered when the player uses punch + a direction at the same time or “side specials” , attacks that are triggered when the player uses special + a direction. These attacks are a crucial part of the Smash Bros gameplay which led to a rewriting of the fundamental design of the code. This code ended up being much more compact and elegant than the early prototype.
Critique & Presentation
Overall, the reception and critiques were positive. One thing to note was that there were two main types of players: those who were familiar with Naruto, and those who had no clue what this was about. Somewhat disappointingly, this crowd largely consisted of the latter: these folks were enthusiastic about the experience and idea, but lacking the nostalgia and excitement of the former. An explained idea, after all, can never replace the power of nostalgia. Someone called the Rasengan “the blue cloud!”, which we thought was funny.
One viewer brought up using this in either a multiplayer video game, or in a table-top game with mini-holograms (referencing Gatebox’s virtual assistant or those pyramid displays) of the characters that would replicate jutsus accordingly.
A player also suggested adding to the UI design of the game, and having what symbols the player was making appear at the top of the screen (to keep track of combos and give immediate feedback). Many other games (especially tutorial modes) do already have this, and we agree it would be beneficial for us to either find one, or code a seperate layer where that is happening on top of our emulator (sort of like the basketball scoreboard from Shilo’s group). This was originally brought up during our development but with the small time frame, wasn’t able to be pushed out in time.
There was one isolated incident where the person had really big arms and we weren’t able to put the gloves on them properly, which was disappointing all around.
One observation we made while play-testing were players’ tendencies to–once figuring out where the triggers were–skip the full hand seal and simply connect the triggers with minimal movement. This, really, isn’t a surprise– gamers consistently find loopholes in game systems to improve their reaction time with little energy (see: moving only the sensored limb in Just Dance, GPS hacking Pokemon Go, etc). For that reason, it’s debatable how much of a design flaw this is and how much of this is just unavoidable without more advanced sensor techniques.
Related works
Flightstick controls for D.Va on Overwatch
This alternative control method follows the same vein of thought as ours: bringing an in-game mechanic/motion to the real world. In this case, flightsticks–matching the D.Va’s in-game flightsticks that she uses the pilot her mech– are wired up to Overwatch as the game console.
One comment points out a flaw: “I know I’m being picky, but it always bugs me that the flight stick input doesn’t match what D.va is actually doing on screen.” This is also an issue that we experienced in our work; the pre-programmed controls for Naruto in Super Smash Flash 2 are limited to punch, kick, grab, and block--not all special jutsus. And there’s no way we’d be able to find a Naruto game that doesn’t have those basic moves that are essential to every fighting game. A solution that isn’t coding one ourselves would be to create an interface that also allows you to punch, kick, etc, along with the special jutsus (adding extra physical controls).
That being said, we would ideally find a more jutsu/combo-based game.
The CaptoGlove Game Controller
This glove-based controller was a good source of inspiration. The glove uses bend sensors and a cool implementation of haptic feedback to create an interesting game experience. On its Kickstarter page, one of the pledge goals was to add a programmable pressure sensor to the thumb of the glove. Unfortunately, they did not make that goal. It is amazing how we have learned to make pressure sensors, in this class, in parallel with a pressure sensor implementation worth over $40 000 USD.
Although we also thought of creating bend sensors with the Velostat, we chose not to because we wanted our players to have free movement of their hands in order to move from one hand sign to the next. The gestures of the Naruto hand signs are a lot more complex than the hand signs of the CaptoGlove. The CaptoGlove also consists of only one hand. We wanted to create a relationship between both of the player’s hands in order to control the game, so we chose to use switches. In order to complete the switch and trigger an action, both hands must be present and meet. It is extremely difficult to find a game controller classic or glove, that requires both hands to make contact.
Nintendo Power Glove
The Power Glove is a glove controller developed by Nintendo in 1989 for the NES (Nintendo Entertainment System.) The player makes hand and finger motions in order to control the game. It uses two ultrasonic sensors to track the roll and yaw of the hand and conductive ink on the fingers to detect how flexed they are. The two ultrasonic speakers on the hand that transmit a signal to three receivers placed around the television set. Through triangulation, it can determine the yaw and roll of the hand. Unfortunately, when the glove came out it ended up being a complete commercial failure due to its poor tracking, and difficulty of use.
Unlike ours, the Power Glove uses wireless ultrasonic technology in order to transmit its data and also tracks hand position rotation, whereas our gloves use switches in order to translate actions to a game. Compared to the Power Glove, our implementation is much more responsive and intuitive.
Future Exploration
As discussed above, a different Naruto game could be explored (Shippuden was suggested). As well, in the critique, different games altogether were suggested.
The different UI/tutorial mode would also be incredibly useful, and improve the gameplay experience by clearly informing you what buttons you were pressing.
Exploring different materials for the build could also improve build quality. As well, soldering wires instead of using the alligator clips; it would make the visible parts of the wires under the guards more aesthetically acceptable.
Further developing it into a more closed product by attaching the Arduino to the arm guard with a battery power source. This would have to wirelessly communicate to the computer either through a direct bluetooth/wi-fi connection, or another Arduino connected.
The initial stage of our project where we constructed a matrix of cubes to build an animation.
Which later progressed into using the matrix of cubes to construct an image from the webcam feed. The farthest we got in this stage was having brief values being read within Toughdesigner but it wasn’t able to create a consistent image of the webcam feed.
Pictured above is the first build of our glove. Initially all the sensors were scattered across the fingertips on one glove but we decided to make two as over time it became difficult to manipulate certain functions.
This was the first attempt at reconstructing our Touchdesigner concept with P5.
Pictured above is the final build of the two gloves used for the critique.
