River Styx (2018)

River Styx

by Shikhar Juayl, Tyson Moll, and Georgina Yeboah

Figure 1

River Styx being presented at the “Current and Flow” end of the semester grad show at OCADU’s Grad Gallery on December 7th, 2018.

A virtual kayaking experience in the mythical word of River Styx. Navigate through the rivers of fire, hate, and forgetfulness using our handmade kayak controller, steering through rubble, rock and ruins. Discover jovial spirits, ancient gods and the arms of drowning souls across the waters between the worlds of the living and the dead.

The project environment was built in Unity using free 3D assets and characters created using volumetric footage. We used the Xbox One Kinect and a software entitled Depthkit in the game:play lab at OCAD U to produce the mysterious animated figures in the project. The vessel is operated with Tyson’s arduino-operated kayak controller, newly revised with 3d printed parts and more comfortable control aspects.

GITHUB LINK

*The scripts and 3d objects used in the project are available via Github, but due to the size of several assets, the Unity project is not included.

Figure 4. Sketched out diagram of circuit.

circuit diagram for paddle controller

PROCESS JOURNAL

Monday, Nov 26

When we first convened as a group, we discussed the possibility of taking the existing experience that Tyson created for his third Creation & Computation experiment and porting it over to Unity to take advantage of the engine’s capacity for cutting-edge experiences. As Unity is more widely used in the game development industry as well as for the purpose of simulations, we thought it would make for an excellent opportunity to explore and develop for new technologies that we had access to at OCAD U such as Virtual Reality and Volumetric Video capture. We also thought it would be exciting to be able to use Arduino-based controllers in a game-development project; a cursory web search revealed to us that Uniduino, a Unity plugin, was made for this purpose.

We also wanted to explore the idea of incorporating a narrative element to the environment as well as consider the potential of re-adapting the control concept of the paddle for a brand new experience. River Styx was the first thing to come to mind, which married the water-sport concept with a mythological theme that could be flexibly adjusted to our needs. Georgina had also worked on a paper airplane flight simulator for her third C&C experiment which inspired us to look at alternative avenues for creating and exploring a virtual space, including gliding. We agreed to reconvene after exploring these ideas in sketches and research.

Tuesday, Nov 27

We came up with several exciting ideas for alternative methods of controlling our ‘craft’ but eventually came full-circle and settled on improving the existing paddle controller. The glider, while fun in concept, left several questions regarding how to comfortably hold and control the device without strain. Our first ideas for the device imagined it with a sail. We then considered abstracting the concept of this controller in order to remove extraneous hardware elements. VR controllers, for example, look very different from the objects that they are supposed to represent in VR, effectively making them adaptive to various experiences and more wieldy. As we continued to explore these ideas, it occurred to us that the most effective use of our time would be to improve an already tried-and-true device and save ourselves the two or three days it would take us to properly develop an alternative. Having further researched the River Styx lore and mythos, we were also very excited to explore the concept with the paddle controller and resolved to approach our project accordingly.

capture

Wednesday, Nov 28

We visited the gamelab at 230 Richmond Street for guidance in creating volumetric videos with the kinect. 2nd year Digital Futures student Max Lander was kind enough to guide and give us pointers about using volumetric videos in Unity. Later that day, we made a serial port connection script to start integrating Tyson’s old paddle script in Unity.

Once that was completed, we then started looking into adding bodies of water assets for our environment using mp4 videos. Turns out the quality was not what we were going for so we later started integrating water assets from the the standard unity packages and began building our scenes.

In terms of the paddle and with the aid of a caliper, we dimensioned the elements of the original paddle controller and remodelled them with Rhinoceros for the purpose of 3D printing. Although the prospect of using an authentic paddle appealed to us, we chose to use the existing PVC piping and wood dowel design in order to reduce the amount of time we spent searching for just the right paddle and redesigning the attached elements. In order to improve communication between the ultrasonic sensor and the controller, the splash guards from the original kayak paddle controller were properly affixed to the dowel, as was the paddle. The ultrasonic sensor essentially uses sonar to determine distance, so it was important that the splash guards were perpendicular to its signal in order to ensure that the sound was properly reflected back. Likewise, we created a more permanent connection between the paddle headboards and the dowel and a neatly enclosed casing for the arduino and sensors.

The process of printing the materials took about five days, as not all printers were accessible and several parts needed to be redesigned to fit on available printer beds based on available material in the Makerlab. We also found that roll of material that we had purchased from Creatron for the project consistently caused printing errors compared to others, which resulted in significant time wasted troubleshooting and adjusting the printers.

capture

Thursday, Nov 29

This was our first session finding Unity assets and integrating them into the Unity editor. We used a couple of references to help shape and build the worlds we wanted to create. We managed to find a few assets we could work with from the start such as our boat. As we continued to find and add more assets to our environment we noticed that some assets were more heavier than others and caused a lot of lag to the game when we ran it so we later decided to use more low-poly rendered 3D models. Once we were satisfied with a certain environment, we added a first person controller (FPS) to the boat which came with Unity’s standard assets and began to navigate the world we created. We wanted to experience what it would be like exploring these rivers through this view and later replace our FPS controller with our customizeable arduino paddle.

Figure x. Shikhar working on River Styx environment.

Shikhar working on River Styx environment.

Friday, Nov 30

Hoping that it would simplify our lives, we purchased Uniduino, a premade Unity Store plugin. This turned out not to be the case, as its interface and documentation seemed to imply that we would need to program our Arduino through Unity instead of working with our pre-existing code developed in the Arduino IDE and its serial output. We ended up resolving this with the help of a tutorial by Alan Zucconi; we transmitted a string of comma-separated numbers associated with the variables that are used to operate the paddle and split them with string-handling functions in a C# script.

After some initial troubleshooting, we managed to get the gyroscope and ultrasonic sensor integrated with Unity by applying rotation and movement to an onscreen cube. The only caveat was that there was a perceptible, growing lag, which we decided to resolve on a later date.

 

Saturday, Dec 1st

As our environment for River Styx’s grew, we continued to discuss the addition of the other rivers involved in the Greek mythological underworld such as River of Pain, River of Forgetfulness, River of Fire and River of Wailing.We then started to brainstorm ideas for map layout, creating a map for these multiple rivers and what reflective element each river should have. Our discussions expanded towards game design vs an exploratory experience. We wanted to see how we could implement certain aspects to make it more of a game and less exploratory. However foreseeing how much time we had to develop and finalize our project without making things too complicated on ourselves we decided to keep it as an exploratory experience.

 

Monday, Dec 3rd

With the continuation of developing out other scenes, we came across other assets to help compliment our indistinguishable river environments. We were able to find lava assets for the river of Fire and create fogs in the river of forgetfulness. With all these assets and possible addition of volumetric videos, we decided we needed a powerful computer to encompass and run our Unity project to further decrease any kind of lag when running and working on it. We considered asking our professors for one but the only one capable of handling our needs was in the DF studio and it was inaccessible for uploading additional software and we couldn’t install any drivers to resolve serial port issues without administrative permissions. To avoid all these bottlenecks we decided to use Tyson’s personal PC tower to continue work on the project and later for installation for our upcoming grad show.  

We also converted the kayak controller code from javascript to C# for use in the unity game engine in an uncalibrated state. The first sign we saw of movement in Unity’s play window was noticeably slow, but indicated that our attempt to translate the code worked. For our convenience, the variables we would need to access to calibrate the device were given the prefix ‘public’ in our code. This allowed us to manually edit them from the Inspector window in Unity without running the risk of adjusting a ‘private’ variable in error.

Tuesday, Dec 4th

We reconvened in the game:play lab to capture volumetric videos with the Xbox One Kinect and Depthkit and import them into Unity. Depthkit comes with several features for manipulating captured data from the Kinect camera, including a slider for cutting out object further than a particular distance and undesirable artifacts. In order to use the captures as looping animations, we tried to keep our recordings in sync with a ‘neutral’ state we determined at the start in order to avoid having the footage jump significantly between the first and last frames. Given that the Kinect and Depthkit render the captured information as a video file we also needed to be mindful about recording times and the number of objects that we wanted to include in our project in order to reduce performance impact.

Some of the animations we captured included hands, exaggerated faces, ‘statues’ of god-like figures and silly dances. We frequently took affordances from the clipped area in order to isolate particular limbs in frame. In one instance, we were able to create a 4-armed creature by using two subjects, one in frame and the other hidden behind in cropped space, contributing a second set of arms.

capture

Wednesday, Dec 5th

At this stage we had three official scenes created. The paddle’s parts were ready to be assembled after going through the laser cutting machine. We then began to create a teleport code where the user could teleport from one cave entrance to the next at each scene but decided not to include it. We wanted the user to explore without feeling the need to be goal driven to get from one place to another. So, we decided to be facilitators and transport them whenever they wanted. We added a key press that would teleport the user from one scene to the next.

We had plenty of fun using the Zero Days Look for our Depthkit captures, which was created for the VR film of the same name. It allowed us to manipulate the default photographic appearance generated by default and incorporate colour, lines, points and shapes into the appearance of the Volumetric renditions. The more we worked with it, the more familiar we became with its interface and how our adjustments would look through in-game, as not all features of the plugin were directly rendered in the world view window in Unity during editing.

screen

capture

Thursday Dec 6th – Friday, Dec 7th

Prior to showcasing our project, we moved all of our unity assets and code to Tyson’s personal PC tower and continued our work from there. We began the integration of the volumetric videos into Unity and  play-tested the environment to get a feel for how comfortable it was to navigate with the paddle. We felt that the kayak’s motion was a bit slow for public demonstration; we tweaked the speed increment, friction, and maximum motion until it felt fluid.

Reception for the project was overall positive. Interestingly, children were able to pick up the controls with relative ease. Since the ultrasonic sensor targeted an area above the size of their hands, they were able to grip the paddle device wherever they desired. This could also be attributed to a lack of preconceptions of how the device works; one of the most experienced paddlers seemed to have the most difficulty operating the device.

capture

 

Project Context

TASC: Combining Virtual Reality with Tangible and Embodied Interactions to Support Spatial Cognition by Jack Shen-Kuen Chang,  Georgina Yeboah , Alison Doucette , Paul Clifton , Michael Nitsche , Timothy Welsh and Ali Mazalek.

 

Tangibles for Augmenting Spatial Cognition or T.A.S.C. for short is an ongoing grant project conducted at the Synaesthetic Media Lab in Toronto, Ontario. Lead by Dr. Ali Mazalek, the team looks into spatial abilities such as perspective taking and wayfinding and creates tangible counterparts to complement the spatial ability being assessed in VR environments. This is all done to explore the effects that tangibles within VR spaces have on spatial cognitive abilities in participants through physical practice and going beyond the idea of 2D spatial testing.

Our project relates to the idea of integrating a customisable controller and using its purpose to complement the situation it’s involved in. For example, in the T.A.S.C project the user is controlling tangible blocks to solve a multi perspective taking puzzle. In our River Styx’s environment, the paddle and its use compliments its environment while also increasing embodiment in virtual spaces. We also designed the paddle in a way that acts as an actual paddle as well. Therefore, If the paddle dips low enough to either the left or right, it will rotate in the direction while also moving forward.

T.A.S.C and River Styx both encompass the exploration of an object’s physicalities and mechanics of a tool and integrate its use in the environment it’s used in. We hope to later integrate VR into River of Styx to increase this immersive experience through paddling in such environments.  

 

Project Context

TASC: Combining Virtual Reality with Tangible and Embodied Interactions to Support Spatial Cognition by Jack Shen-Kuen Chang,  Georgina Yeboah , Alison Doucette , Paul Clifton , Michael Nitsche , Timothy Welsh and Ali Mazalek.

Tangibles for Augmenting Spatial Cognition or T.A.S.C. for short is an ongoing grant project conducted at the Synaesthetic Media Lab in Toronto, Ontario. Lead by Dr. Ali Mazalek, the team looks into spatial abilities such as perspective taking and way-finding and creates tangible counterparts to complement the spatial ability being assessed in VR environments. This is all done to explore the effects that tangibles within VR spaces have on spatial cognitive abilities in participants through physical practice and going beyond the idea of 2D spatial testing.

Our project relates to the idea of integrating a customizable controller and using its purpose to complement the situation it’s involved in. For example, in the T.A.S.C project the user is controlling tangible blocks to solve a multi perspective taking puzzle. In our River Styx’s environment, the paddle and its use compliments its environment while also increasing embodiment in virtual spaces. We also designed the paddle in a way that acts as an actual paddle as well. Therefore, If the paddle dips low enough to either the left or right, it will rotate in the direction while also moving forward.

T.A.S.C and River Styx both encompass the exploration of an object’s physicality’s and mechanics of a tool and integrate its use in the environment it’s used in. We hope to later integrate VR into River of Styx to increase this immersive experience through paddling in such environments.  

The Night Journey by  Bill Viola and the USC Game Innovation Lab in Los Angeles.

The Night Journey is an experimental art game that uses both game and video techniques to tell the story of an individual’s journey towards enlightenment. With no clear paths or objectives and underpinnings from historical philosophical writings, the game focuses its core narrative towards creating a personal, sublime experience for the individual participant. Actions taken are reflected upon its world.

The techniques incorporated from video footage and the narrative premise of the game gave us inspiration for how we might tackle the scenic objectives for our project and interpret the paths we wanted players to take in River Styx.

References

  • https://docs.unity3d.com/ScriptReference/Rigidbody-isKinematic.html
  • https://forum.unity.com/threads/simple-collision-detection-blocking-movement.86712/
  • https://docs.depthkit.tv/docs/unity-plugin
  • https://www.unity3dtips.com/move-objects-in-unity/
  • http://www.uniduino.com/
  • https://www.arduino.cc/
  • https://www.zerodaysvr.com/