Author Archive



Underwater Dream is a game that can be played by your mind. The purpose of this project is to create dream-like experience of underwater.

This idea came from my own dream experience. When I was dreaming, I can experience amazing story that never happened in real life. If I realized I was dreaming, I can control the dream by my mind. For example, if I want a cake, I can imagine there is a cake in my hand and after I blink my eyes, I will see a cake in front of me, like magic.

So there are some points that make dream experience different from real life:

1. Without real activity

2. Change surroundings by imagination (and blink)

3. Fantastic story, but looks real

Because many unexpected physical errors appeared (and difficult to be fixed) in Project 1, I don’t want to make many physical instalments. So this time, I chose game, which is a good way for interaction, as the platform.

To decide what element can be used for water park, I had a brainstorming:

Screen Shot 2014-11-17 at 7.00.50 AM



To achieve these points, this game combines three different data sauces: EEG sensor, Arduino, webcam.

EEG sensor is used to achieve point 1 and 2. Player can go forward by attention level, play music by meditation level, and destroy barriers by blinking eyes.

Arduino is used to connect water in real world to virtual water, by detecting water level.

Webcam is used with openCV to detect player’s face, to make the world in screen looks like a “real” one because player can have different view angle by moving head.

Here is a reference video about the “real” 3D effect:

To summarize, this game can be played without keyboard and mouse, only mind and motion.


About the scene, there are two plans:

First, the whole scene is a circle to make the game be able to circulate. Player can turn automatically. There are three themes for the whole map: Coral Valley, Sunken Ship, and Underwater Graveyard.

Second, the scenes are straight path that player can not turn. There are two themes for two scenes: Life and Death, player starts the game from Life scene, in which all the sceneries are light and lively. After player experiences the whole scene, he/she will be transported to Death area, which is full of dark and horrible skeletons and remain of sunken ship. After this, player will be brought back to Life area to complete a circulation.



         Unity 3d


3Ds max



         openCV for Processing





The main challenge is how to transfer data between different sources. The data come from Arduino, Webcam, EEG sensor to Processing, and then transfer to Unity.

Screen Shot 2014-11-17 at 6.42.26 AM


From Processing to Unity

First of all, I have to make sure Processing can be connected with Unity. I found a video that let Processing control the scale of a cube in Unity. As the video said, I download the library called oscP5, with this, the number can be sent to Unity successfully. This is the base of the whole project that should be confirmed before everything start.

From Arduino to Processing

I use ultrasonic sensor in this project, to detect water level in a box. The game will start only the box is filled with water.

Here is the circuit:

Screen Shot 2014-11-15 at 9.45.25 PM

Here is an experiment, for testing if the ping sensor can detect the water level:

//Experiment 1

(Some description of experiment is not enough 200 words because I didn’t write in order, some descriptions are in other section, and also some descriptions are for more than 1 experiment)

Here is another experiment for testing the data transformation, from Arduino to Processing to Unity.

//Experiment 4

From EEG to Processing


The first time I knew EEG is, I watched a TED talk about Emotiv, I was so surprised to the technology (and this is an important reason why I chose Digital Future). I have a NeuroSky Mindwave EEG sensor, but because I don’t have background of coding so I didn’t use it until I learnt Processing. There is a library for EEG sensor called ThinkGear, and fortunately, NeuroSky company provided lots of develop tools in their website and some of them are free.

To transfer the data from EEG sensor to Processing, two software are necessary: MindWaveManager and ThinkGearConnector. The first one is used for connecting EEG sensor to laptop; the other one is the socket that can help user to transfer data from sensor to Processing. Both of these can be download from the website.
There are several variables that have been provided in the library: attentionLevel, meditationLevel, blinkStrength, delta, theta, low_alpha, high_alpha, low_beta, high_beta, low_gamma, mid_gamma. The level of these can reflect user’s emotion. In this project, only attentionLevel, meditationLevel and blinkStrength are used.

This is Experiment 2: Data from EEG sensor to Processing

In this experiment, I used attentionLevel to control the height of this rectangle, and the blinkStrength to change the color of it. The harder I blink my eyes, the more vivid the color is.

//Experiment 2

Here is Experiment 5: using attention to control the movement and move speed. Player can move faster by concentrate more, and if player lose concentrate, he/she will stop.

//Experiment 5

In Experiment 6, the blink function has been added. When my strength of blink reached a certain level (it can avoid slight blink disturbing the result), the barrier disappears. This function is used to stimulate the situation in dream that I mentioned in the first part. The advantage of this function is the player won’t find the process of disappear because it happened when the player close the eyes.

//Experiment 6

From Webcam to Processing


For tracking face with webcam, open CV library is used. From the example of face tracking, I tried to locate the middle point of face, and use the position to control the camera in Unity. The final effect is, when you move your head, the camera in Unity will move and rotate with your movement.

//Experiment 3

Work in Unity

After transfer data from Processing to Unity, the other works were completed in Unity.

The first challenge is Processing is not supported in Unity so I have to use JavaScript, which I never used before. I completed the code of JavaScript by learning from examples, online tutorials and official Unity manual. From learning JavaScript I found that even JavaScript and Processing are different languages, they are still have something similar that the knowledge I learnt from Processing is helpful for learning JavaScript.

In Unity, I move the whole scene instead of moving the character. In Experiment 3 I already tested the camera movement function, in Experiment 5 the movement function has been tested too, and in Experiment 6 the basic gameplay was completed. But they all relate to Processing. Here is the function just effect in Unity.

This experiment is an improved version of Experiment 6, because I need this game is automatically circulate without stop to restart, so the barriers should appear again. The Experiment 7 is for this function. The effect is, after the second barrier is destroyed, the previous one will appear to the original position.

//Experiment 7

Work in Processing

Experiment 8 is for background music that can be controlled by meditation level. When meditation level over 50, a piece of music will be played. It uses a library called minim. The music will not play a second time until it finishes for the first time.

//Experiment 8

Experiment 9 is for testing objloader library. I tried an example but when I tried to import my own obj, it can’t show the texture and I don’t know how to solve this problem. Objloader is another choice if Processing can’t transfer data to Unity. But obviously Processing can’t support 3d as convenient as Unity.

//Experiment 9

Experiment 10 is the one I tried to let player turn automatically. But unfortunately I just complete half and because at first I didn’t set the original position properly so it’s difficult to change it. I tried 2 methods to trigger the turn, the first one is use distance but it’s not accurate. So I tried to use collision. This one is much better but because I don’t have enough time to modify the position, I didn’t add this function into my final version. But I forgot to record this part….



Processing and Unity:


The 3d models were made in 3ds max, the texture were drawn in Photoshop.

Here are some reference pictures:

1 2 3 4

Here are some pictures for modelling:


Import 3d models to Unity:


Also I added many effects such as light, fog, particle and so on in this scene.

4. Conclusion

Here is the final version of the project:


Actually this is not a project for showing to publish. It concentrates on personal experience. But I think a multi-player version will be more attractive. And also because of the time, I didn’t complete all the things I want to show.

From this project I really learnt a lot. This is the first time I made something about code. I am so excited about learning a programming language (even I met a lot of difficult, both in Processing and JavaScript, and some of them haven’t been solved now). But I think coding is very important in a project because I found many ideas cannot be realized because the lack of program knowledge.

Project_01 Trapped Soul




The name of this installation is Trapped Soul.

This work is an installation that use strengthen of muscle to control a “soul” escape from a special container.


Background Story

This scene happens in an abandoned space ship. 50 years ago, there was a research about artificial life. The scientists launched a space ship with an embryo. But unfortunately, the embryo grew to an alien because of the cosmic radiation. All the scientists on this space ship were killed by it. The leader of this scientist team is the one who survived from alien’s attack, but people on the earth did not allow him back because they afraid alien will come to the earth with him.

At his last moment, his still did not want to kill the alien. He spend thousands days on it and regarded it as his own child. His feeling was so strong, so after he dead, his soul become a spirit embryo that trapped in the embryo container, waiting for someone letting his soul get free.

Design Plan

The purpose of this work is letting people experience a magical moment that they can control a ball’s movement, also the brightness and sound.

With participant tenses the muscle, the “soul” lift up, changing color from white to red, and when participant relax the muscle, it dims gradually and goes down. It will burst out a scream when participant tenses the muscle with full power.

To make the scene more immersive, I plan to build some pieces of a dead astronaut, when people get close, the hand will shake to call the people and a box will open to show the fabric pad.

But because of the time, I didn’t finish all the content, just built the essential interactive part.



  • Transparent plastic board *1
  • Tapes
  • Transparent ball *1
  • A roll of fishing line
  • Elbow pad *1

It includes two main parts, one is a large transparent plastic tube with an Arduino UNO, a motor shield, a motor, a speaker, a ultrasonic sensor and a muscle sensor with two 9V batteries.

Another part is a fabric pad that can stable the electrodes of muscle sensor on one’s arm. It’ s reusable and provides a more accurate position for electrodes.







Muscle sensor can detect electric potential to measure muscle activity. When it is connected to Arduino, it can output a number depending on muscle activity. Typically it’s 0 when the muscle is totally relaxed and the harder you tense your muscle, the higher the number is. This number controls the motor, speaker and LED.

First, The number is used to control the direction of motor to lift and down the LED ball. When you equip this fabric pad on post brachium, if you tense musculus biceps brachii, the motor rotates to lift the ball, and when you relax the muscle, it rotates oppositely so the ball drops down slowly.

Second, the LED is linked to the muscle sensor. The number maps to the brightness of the LED between 0~255 from dim to bright, so the harder player tenses the muscle, the brighter the LED will be. When the number is 0, the LED extinguishes.

Third, the tone of the speaker is also linked to the muscle sensor with map function. Also the harder player tenses the muscle, the tone become higher.

This is used to create a magical atmosphere because people can move the ball and light it using original energy without touching it.

In addition, I added some function to avoid error in this system. The first is added denoise code to filter noise value. The second is I used a ultrasonic sensor to measure the distance between the ball and the bottom of the tube to avoid rotating oppositely.




  • Arduino UNO *1
  • 100 ohm resistor *2
  • Motor *1
  • Ultrasonic sensor *1
  • LED *1
  • Speaker *1
  • Muscle sensor *1





I changed my ideal for several times and now the Trapped Soul is the final version.

First, I planned to make a scary box, using light sensor to trigger an action. When people look into the box from a hole on it, the light sensor can detect the change of the light, and then trigger a servo or a motor to pop out a fake hand. This is a basic idea and I want to add some LED in it as the stars in universe to tease people look into the box and keep watching it until the hand appears.

7509c7d0f703918f9f0a3abe523d269758eec49d b2ecd81ed21b0ef4ffc89e2adec451da81cb3e54

I spend several days to prepare the materials and try to let it move properly, but after I made this, I found it didn’t look like what I want, the box is too small to hide the hand and it looks not scary and not technical at all. So I decided to give it up and make something else.

At that time, there’s only less than 1 week before deadline but I haven’t decided what I will do. I have another idea about a dead astronaut and move it by servo but it’s all about the visual impact so I still unsatisfied about it. I want to build a installation with strong interaction with visitors, not just let them look something happen when they passed by.

When I thought about interaction, the first thing that pop into my mind is muscle sensor, which can be used to trigger activity directly. I had a brainstorming, listing every element related to spaceship and came up several ideas. At last I chose anti-gravity as my theme.

Then, I started to learn how to use muscle sensor to show anti-gravity.It’s obvious that letting something float in the air with out any support can remind people anti-gravity. And this project also related to haunted house, so a soul is a perfect theme to combine both anti-gravity and haunted house.

After I decided the theme, I started to learn how to use muscle sensor. It’s totally new for me. I found some tutorial by google it but not too much. I wrote a very simple code to print the output number for testing, and then I met my first challenge also the hardest one: there is no output except zero. I spent two days to test it. I googled the problem but only found many people have the same problem, without any solution. I measured each connection joint of battery, the board and electrodes but found nothing wrong with them. I was so anxious because it’s near the deadline and I haven’t started anything even the most basic function.20141003_013231

After struggling for 2 days, I finally found out why it didn’t work. The reason is I am not strong enough to active the sensor effectively. My friends helped me test it and I wrote down the number for triggering other functions.

The next step is letting the motor can rotate both directions. I used motor shield to control the direction and speed. Here is a video for the first success that using muscle sensor to drive the motor. I ‘d like to share it but I don’t have enough time to edit and upload to Vimeo.


These are the steps that I change the material and test the motor.

First I cut a  foam ball as a wheel and use a lantern as the soul, but both materials are not good.



Two days before deadline, Gary found a plastic ball for me. It’s much better than the lantern.



At the same day, I replaced the foam wheel by the frame of fishing line.



The last day, adding the speaker, and then the tube for decoration. This idea was also came from a brainstorming with Daniah and Gary.



The first challenge is no output from the muscle sensor. I tested it hundred times to find out why it cannot work. What I learnt from this experience is, the position of  electrodes is really important, so that’s one of the reason that I want a fabric pad to help locate the electrodes, ensuring the electrodes can be kept on a certain position every time.

Second, I met another difficulty when I tried to use RGB-LED to change the soul’s color from white to red, but It didn’t light when I connected it in this circuit. This problem still hasn’t be solved now.

The third challenge is about the material. It’s difficult to find required materials and some of them are impossible to make by myself. So the visual effect didn’t reach my expectation.

The last one is about the wire and fishing line. They are too easy to twist on the motor and because of the tube, it’s extremely difficult to fix it.


Next Step

Actually at last I still not very satisfied about my work. To solve some existing problem and make it more attractive, it can be improved in these aspects:

  1. Decoration: Adding a dead astronaut next to the tube, and 2~3 led strips connecting the tube and the astronaut. Light the led on the strip one by one to simulate the soul flowing to the container from his body.
  2. Wireless: Comparing with the pros and cons of the brightness changed led, I found if this function is canceled, the whole system can be much more stable because there will be no wire disturb the fishing line and the motor. But another plan is, using wireless method to control the light, just put a electricity source in the ball with the LED. Maybe Xbee can be used in this step. (but I never used it before)
  3. Gamification: If it’s possible, I want to make the whole project as a mini game, with more interaction with visitors, including a complete background story for people engage in, more trigger for surprise and people’s action can lead to different results. It can be designed as an electronic game but with physical installation and more immersive.


The first idea is come from the movie called Alien, choosing the idea of popping out something from a container. But I changed it a lot. The second idea appeared in my mind in the process of brainstorming without any reference. But I found some similar project using muscle sensor and also fabric equipment like the two links below:

This is a project that using a hand-made muscle sensor to control the frequency of sound when people do sports

This is a tutorial of how to make a conductive fabric electrodes.


Use of this service is governed by the IT Acceptable Use and Web Technologies policies.
Privacy Notice: It is possible for your name, e-mail address, and/or student/staff/faculty UserID to be publicly revealed if you choose to use OCAD University Blogs.