The Soundscape Experience

The Soundscape Experience


Tyra D’Costa | Shiloh Barnes | Francisco Samayoa

screen-shot-2018-11-26-at-6-19-54-pm

Project Inspiration

Our project seeks to encompass elements of sound, sensory perception, and nature in a installation based artwork. Our process began by discussing the types of sensors we wanted to explore, eventually we decided on the MPU 9250 Accelerometer and Gyroscope. Our proposed idea, is to integrate the sensor into headphones, which can then change the sound and visual experiences of the user. As the user navigates the room and moves their head about, the world in which they are digitally immersed in will enhance the real world.

Goals

  1. Engaging UX Design: We want to solidify the conceptual ideas of the soundscape experience to create an immersive UI/UX design.We want this to include, elements of storytelling and gamification.
  2. Sound-Sensory Experience: Moving forward we will have to test that the sensor can allow smooth transition between sound files. The media assets will consist of ethereal soundtracks, data visualizations (natural algorithms and patterns), and nature related images.
  3. Integrate wearables: Also, we need design a way to integrate the sensors into the wearable technology ( the headphones).
  4. Discuss concepts of nature, ecosystems and natural algorithms: Lastly, we want to think about how these concepts can work together to create a narrative and gameplay element.

Background Work

So far, we have able to get the MPU sensor working with Arduino so that we receive X,Y,Z coordinates. This required, scouring the web for solutions to library related issues, and calibrating the code in order to receive accurate data readings. Additionally, we were able to take these coordinates and map them to a camera in Unity, so that the sensors orientation changes the perspective of the viewer.

Links To Contextual Work

How to play multiple audio files

Research on Arduino + Gyroscope

Good Link explaining how IMU sensor works

Tutorial for IMU and Unity set up

How to set up MPU and Arduino Library

Unity + Arduino Library

Project Details

  1. User starts at any location on the map
  2. They put on the headphones
  3. Depending on the position of the head, the headphones will play a particular sound.
  4. If a particular sound is playing then the screen will display an associated image or video
  5. The user must experience all the audio and visual data in order to try and make a connection between the media and a place that exists in the real world before their opponent. 

Sketches

img_4916img_4924img_4924img_4926

Process

46837114_564283070696082_6498849018557235200_n 46812922_473012876557775_1587641447014727680_n

46801407_337187233779982_3353290089944842240_n 46854703_644056072656401_7774253524338081792_n 46796952_337902460325535_4724015759863316480_n

Blast From The Past Arcade

 

screen-shot-2018-11-19-at-1-36-18-pm


Tyra D’Costa | Ola Soszynski | Kiana Romeo | Dimitra Grovestine

Project Description 

For the third experiment, our team studied a variety of conductive materials that could be used as conductive sensors. We applied these sensors to wearable based controllers, in order to create a larger purpose for them in terms of their use in gameplay, and design for health. Our process began with a discussion on a topic we all found nostalgic- our childhood. We all remembered playing  Miniclip and Disney games when they first came out. Being one of the first generations to grow up with technology, this topic seemed both fascinating and relevant to us. As we reminisced on our favorite childhood games, we also began to talk about the very real effects they had on social health of the millennial generation. Many of us remember rejecting the park or our friends to stay inside and play computer games or watch Netflix. Soon after, social media would further this drastic change in how children play, interact and socialise.

screen-shot-2018-11-19-at-1-39-12-pm

We also discussed some of the stereotypes and harmful effects associated with gaming for long periods of time. These effects may include:  anti-social behaviour, stress in the tendons of the hands and forearms, reduced mobility etc. Our focus was to think about how we could reintroduce the games we loved so much as children, while addressing some of these issues.  We wanted to create buttons that either engaged a group of individuals together, or encouraged an individual to engage in physical activity. However, we wanted to ensure that these purposed engagements did not have negative longterm effects on the body, and that the controllers were ergonomically designed to  enhance the users experience.

img-4901

Materials and Design

Using a Makey Makey micro controller eliminated the need for any code based work,  we really wanted to focus our attention to building and designing the conceptual aesthetic  of our project. Instead of making a single game controller we decided to make serval and host a 2000’s themed arcade. This involved analysing various elements of the games we chose. We observed the movements, directions and positions of each game in order to design controllers that were intuitive to the player and met the standard of our design goals.Our team concluded that it would useful to replace the traditional controller with wearable controllers in order to optimise the fluidity and interactivity of the designs. We thought it would be fun to make the controllers funny outfits from the early 2000’s, so we created mock ups of potential outfits and brainstormed design ideas for how they could be transformed into game controllers. The conductive thread became very useful when we wanted to hide wires and create a more seamless look.  To contribute to the overall theme of childhood fun and nostalgia we decided to add prizes and a ticket booth. This added a whole new dimension to the arcade, bringing a sense of good spirited competition and challenge to the user experience.

Design Goals 

  • Economic Design
  • Addresses social issues
  • Design for health
  • Interactive user experience
  • Kinetic user experience
  • Childhood Fun and Nostalgic Themes

Materials

  • Used clothing ,
  • Makey- Makey micro controller,
  • Iron on conductive fabric,
  • Conductive thread,
  • Aligator Clips

Outfit Mockups

2003 - Rocker Look
2003 – Rocker Look
2006 - The Dude
2006 – The Dude
2010 - The Hispter
2010 – The Hipster
First Sketch of the Arcade User Interaction Design
First Sketch of the Arcade User Interaction Design

How it all Works

In the Blast from the Past Arcade, every game we had used the Makey Makey, a small board that works like a USB and sends WASD/ left, right, up, down and spacebar signals to the computer. When the circuit is closed for any of these parts of the board, it sends the command to the computer being used as if the button was pressed on the keyboard.

Makey Makey Pluggeded into our Happy Wheels game contollers.
Makey Makey Pluggeded into our Happy Wheels game contollers.

Using the Makey Makey was quite easy since when connected to a computer it takes over the keyboard controls that would have been used otherwise. This allows for more movement in playing video games because the player is not required to sit directly in front of a screen and push buttons, they can get up and move around depending on how it is hooked up. As seen in the photo of the Makey Makey below, it can be seen how simple alligator clips and wires at various lengths can be attached in order to make more interesting controllers.

Makey-Makey Reference page

When performing the first test for the Makey Makey, we used it on the games we performed our research on, games that required an assortment of keyboard controls. The first game we tested was Super Mario Bros as this is a classic keyboard/ controller game. 

Video Documentation – Learning to use the Makey-Makey: Our First Success

Finally, when we put together the games, we had to ensure the controls on the Makey Makey corresponded with those for the specific game. For example, with the Sandwich Stacker game, the “left” and “right” arrow controls on the Makey Makey were connected to the two players wearing the suspenders and glasses while “ground” was connected to the player wearing the bow tie. Therefore the person who was grounded would simply have to make skin contact with the other players to activate the left and right controls.

glasses controller ( far left ), Bowtie contoller (middle), and Suspenders contoller (far right) in use.
glasses controller ( far left ), Bowtie contoller (middle), and Suspenders contoller (far right) in use.

With regards to the Club Penguin Dance Game, the player was connected to ground on each of their hands through the edgy gloves. The player wore a rock’n’roll t-shirt with two conductive iron on shoulder pads. These two shoulder pads were connected to the up and down arrows. Finally, the side movements were controlled by iron on conductible fabric on the hips of the belt. The gloves, not only were grounding themselves, but they also grounded the entire human body. Prior to getting the gloves to work, we had tested out using a single wristband. Because the wristband grounded the entire body, players could still use either hand to complete the switch and produce the proper game movement.

Edgy gloves is used as a ground to complete the circuit.
  Edgy gloves is used as a ground to complete the circuit.
The full Edgy Club Penguin Dance game controllers outfit
The full Edgy Club Penguin Dance game controllers outfit

For the Happy wheels game we  made two floor based controllers modelled after car pedals, we thought this was an inherently intuitive design for the acceleration and break.  If the user is wearing the conductive ground wrist bands they can touch the left and right sides of the hat, as well as the centre of the bandana to front flip, back flip and reverse. While designing these game controllers we paid close attention to our research to ensure that the design was safe for long periods of game time. We made sure to keep the neck aligned, the bean bag chair came in handy for this as we were able to adjust the game screen as needed. Secondly, the design allows the elbows to always be bent at either sides of the player when playing. Lastly,  the hand of the player is always in the same plane as the forearm which is key to reliving stress points in the hands and tendons.

Game controllers for the Happy Wheel Game
Game controllers for the Happy Wheel Game
Floor pedals for Happy Wheels game
Floor pedals for Happy Wheels game

Research

When conducting research for our Blast from the Past Arcade, we had to consider many factors. Firstly and possibly the most important part of our project we had to carefully select games that we would use within it. These games had to have certain criteria:

Nostalgic games from different eras of our childhood:

We had to ensure that the games would resonate with everyone who tested our arcade and also with us in particular. We tested many games and had to dig deep in the internet to find these games (Some of them are over a decade old!) furthermore, since the arcade was about revisiting the past, we had to exclude some of our beloved games as they coincided with the same era. Research on the game release dates was necessary so we could successfully spread our arcade across many years (around 2005- 2012).

Games that use any combination of keyboard controls (spacebar, left, right, up, down buttons):

Although we had many games in mind, the Makey Makey requires keyboard controls to work. Therefore, a game with mouse movements would not work at all while a platform game using the left and right keys would. Furthermore, while researching, our group decided that instead of looking for games that could potentially use mouse movements, it was more authentic to use games with keyboard controls.

Links to games we had tested with the Makey Makey (games highlighted are the ones we used in the final prototype):

Ergonomics

The next step in our research was to figure out the science and implications of the way movements of the body affect our health and also how different game controllers and the way we hold them effects game play. The most logical way to go to find info on this was to look for research conducted by e-sports specialists. Our research found that posture and ergonomics are directly related to successful game play. Certain postures and movements provide the body with relief from cramps and muscle pain. Furthermore, when playing video games, watching the screen is a very important part of the experience but it’s also known that staring at screens for long periods of time can strain the eyes. By making the placement of the game eyelevel and the screen at least an arm length away from the player, the gaming experience will be a lot better. As well, wrist and finger positioning should be in such a way that it makes it easy and intuitive for someone to use.

When we designed our interfaces, we took all of these facts into account. For example, with the Happy Wheels game, one would sit on a comfortable bean bag in which they could position it to their liking. The controls included intuitive foot pedals with one used to step on the gas and move the character on the screen forward and one used to break. This freed up the players hands so that they could reach up to the hat and press it to control certain things in the game and lastly the bandana on the chest of the player one would simply touch to activate. We thought carefully about the controls with all of our games, both relating the functionality with the theme of the game itself and making sure the controls were comfortable and ergonomic enough to make playing more fun.

Research sources:

http://www.1-hp.org/2016/10/28/esports-health-it-starts-with-ergonomics-and-posture/

https://medium.com/what-the-tech/ergonomics-of-gaming-gear-a07058f88bf3

Documentation

Click to see Club Penguin Dance + Rocker Look

Activates the body and encourages users to move and dance in order to gain pints, win tickets and get prizes!

Creating the 'Edgy Gloves'
Creating the ‘Edgy Gloves’

Click to see Happy Wheels + The Dude

Encourages the user to step outside of the traditional hand positions found in most game controllers. This design relives stress that could build in the the tendons and muscles of the hands.

Creating the floor pedals
Creating the Happy Wheels floor pedals

Click to see Sandwich Stacker + The Hipster

In this game users have to interact and work as a team to keep the game going, beat high scores and win big!

Wiring up the Sandwich Stacker game with Conductive Thread
Wiring up the Sandwich Stacker game with Conductive Thread

*Click to See the Full Arcade Experience *

A group of 3 working together to set a high score on sandwich stacker
A group of 3 working together to set a high score on sandwich stacker
Nick all ready and wired up to play club penguin dance.
Nick all ready and wired up to play club penguin dance.
Explaining the how to use the game controllers.
Explaining the how to use the game controllers.
Ola explains how to use the game controllers for the sandwich stacker game
Ola explains how to use the game controllers for the sandwich stacker game