MVP Console

MVP Console


Project Description

MVP Console is an interactive installation based on the tangible interface experience. Players are invited to play with three digital activities created in Processing: Core ball, Spinning Music, and Starry Drum. MVP Console is a handmade cardboard box with 5 aluminum foil buttons that attached the 5 pins from MPR121 touch sensor. In this project, MVP Console stands for Musical, Visual, Physical Console; it makes the players enjoy physical touching, music creating, and music visualization. The players could switch the card paper upon the box to play the matching activities.

The first activity is Starry Drum. Once the player touches the buttons, they would hear constant drum beats and the starry dots would appear on the screen. Every button represents a different part of the drum and the different sizes of the starry dots. The dots would fade away once the new dots appear. In this activity, the players are creating an image of a starry sky and play with the drum at the same time. 

The second activity is Spinning Music. The players could touch the button to play the songs and watching the spinning pattern, which is the music visualization. There are 5 songs overall, the spinning pattern would change color and speed that matches for each song. The songs I used are Woodkid’s music: “L’aérogramme de Los Angeles”, “Iron (Quintet Version)”, “I Love You”, “Iron” and “Run Boy Run”. The spinning pattern and song would stop once the player releases the button, it is a unique way to touch with music and visuals.

The last activity clones a popular phone game, Core ball. The players need to shoot the given needles to the rotating circle. Every shot needle would be stuck and rotate with the circle. If your needle hits an existing needle on the circle, it will be automatically judged lose by the system, you need to restart to continue. The screen would turn blue when you win the level, and red if you lose the game. The players need to press three aluminum button that labeled C, S, and SHOOT! to play, everything is drawn and labeled on the card paper. The players can see their level on the screen, and the number of needles left in the center of the circle. The higher the level, the more needles you need to shoot. 

Experience Video

How it works

Final project images






Development images


img_2306 img_2309


Link to the Arduino code hosted on Github

Circuit Diagram



Project Context

MVP Console is inspired by my favourite entertainment and learning device in my childhood, an electronic dictionary. It is like a small size laptop, it was heavy but convenient. Other than look up the vocabulary on it, it is my “all-purpose media player”. It could be the music player, and once the music is playing, the screen would show the music visualization, which is the spinning firework pattern. Although it has a dull green dot-matrix screen and it has only one song, its music visualization was like magic to me in my childhood. It is also my game console, it has one game, Snake, which always made me lose time track. Finally, it is also my musical instrument. Every button has a different sound effect, I used that to cover all popular songs in that period, I even used that to perform in the music class. The electronic dictionary is now obsolete, its functions are simple, but it gave me countless happy memories. I want to recreate my childhood memories.

Once I know the theme of Experiment 3, Tangible interfaces, the first thing pop into my mind is about music. It reminded me of the tap-tap games, they have the perfect combination of touching, visual, and music. Other than that, it also reminded me of the electronic devices that could imitate the sound of real musical instruments, which could be a more convenient way of composing music. It could also be about music visualization, combining the physical interaction with the seem music could be an inspiring experience. Music is important in my life. It affects my emotion, and it is completely connected with my vision. I composed with my vision and I listen to create images.

For this project, I want to explore the relationship between music, visual, and physical. I want to make an “all-purposed media player”, an “old game console”. Inspired by Tyler Crumpton’s Capacitive-Touch Arduino Keyboard Piano, I want to make the button with aluminum foil, tapes, and cardboard as my material. I made a cardboard box with 5 square holes on the top, and I sticked aluminum foil under each hole. There are three card papers with my drawing and holes that fit each game.  The players could switch the card paper on the box, like switching the disk of the game console. The touch of aluminum foil buttons is the Physical part of my MVP console. 

I used the sound effect of the button from my electronic dictionary to make music. Even the musical instrument are not with me, I could make music. It is kind of like the calculator cover music, which is now very popular. It is a creative new way to make music with unconventional methods. Each button of the musical calculators corresponding to a musical note. With two calculators, the players could try the classic piano-playing method of using both hands. (Lee) It is quite inspiring, as to play music with 5 buttons, I want to recreate it with the sound of drumbeats. When I was learning drums, with the practicing with the speed of beating, the countless beats in a second make me imagine a thousand-star sparkle, the stars blink with the rhythm. I want to combine those visual imaginations with the sound of the drumbeat. I purposely make the drumbeat unstoppable if the touch isn’t released, there is no single beat. To fit the speed of the drumbeat and the endless stars. Repeating the same sound could be beautiful, same as visuals.

Spinning Music is a musical visualization of the rotating pattern. The looping rotating pattern always makes me lost the time track for just staring at it. After watching the tutorial of the rotating pattern on Youtube, I think it could fit the music well, it could be the animation of music visualization. By changing the speed and color of it, it could work with music in different styles. That is one of the reasons I chose Woodkid’s music, he has various styles. His songs get to my emotions and speak to me a lot. The visual and the music reflects what he states “I like to translate sounds into images and images into sounds”(Dellarciprete) The pattern would keep rotating when we touch the button, when we stop it, even the pattern is static, we would feel like it’s rotating backward. It creates an optical illusion, which also indicated the melodies was stuck in our minds. Music affects our mood, and we are exploring 5 different emotions and memories in our mind in this activity. 

Coreball is a popular phone game, I fit it with the shooting sound effect. As there are 5 buttons overall and we only need 3, so I cut  3 holes on the card paper. After we are familiar with the game, we could form beautiful patterns, like the dandelion. The game rule is simple but quite addictive. Every shooting needle combines the human touch and sound effect. 

Overall, with the topic of the tangible interface, I make the viewers explore 3 types of touching. In Core ball, we used a single click to have one shoot and sound effect. In Starry Drum, the drum sound is constantly beating after we touch the button, we would not hear a single beat. In Spinning Music, the music start when we touch it, and we need to hold the button to keep the music playing, it would only stop when we release it. I want to make it like a real game console, with only a few buttons, but you can play multiple games. 



Crumpton, Tyler. “Capacitive-Touch Arduino Keyboard Piano.” Instructables, Instructables, 23 Oct. 2017,

Dellarciprete, Alex. “Woodkid’s The Golden Age – An Analysis of an Audiovisual Experience.” Medium, Medium, 12 Dec. 2017, 

Lee, Dami. “Calculators Are the Latest Instrument Used in Creative New Ways to Make Electronic Music.” The Verge, The Verge, 11 Oct. 2017, 




Team: Jay Cooper, Bernice Lai

Project description

NINE is a remote asynchronous messaging system that uses LEDs to codify communication between two users. LEDs are arranged in a 3×3 grid and activated using touch sensors. Users can touch the surface near an LED to activate the LED on their partner’s device while turning off their own. This signals to the partner “I have received your message”, creating a constant back and forth between acknowledging each other’s messages.

NINE is like a “mood board”. The users are sending their present state with LED. They can then “acknowledge” the other person’s mood by turning off the light to marked as seen.

Experience Video

How It Works Video

1 2



Final Images








microsoftteams-image-4 microsoftteams-image-3

microsoftteams-image microsoftteams-image-1_%e5%89%af%e6%9c%ac



Link to the Arduino code

Circuit diagram


Project Context

NINE is an interactive project for long-distance communication; it combines the LED and physical touching to build interpersonal communication. We use light as a signal to messaging each other our moods. The remote presence overcame the distance to know the mood of their friend synchronously. The LED in 9 different positions represents 9 different moods, NINE is connecting people over distance and time zone.

Because of the Covid-19 pandemic, people are using more social media, email, phone texting, etc, to communicate with each other, the way people express feelings and emotions through the physical environment is weakened. Since the home is a safer place to stay in and we cannot meet up with our friend, remote presence should be developed more as it is a good choice to stay at home, even in another city, another country. 

For making this project, we are inspired by the friendship lights and IoTConnected Chess Boards. Both of them are physical interaction with lights. Friendship lamps are designed for a long-distance relationship. A simple touch could change the colour of the light and turn on the lights on the other side to signal the person. The lamps are connected to the wifi, it could interact with your friend or lover in real-time even if they are far away. Turn the lights on and change the colour of the light on the other side is a beautiful representation of telling us we are thinking of each other.

IoT Connected Chess Board is an electronic chess board that connects to Wi-Fi. Each of the users would get the board with the pieces of one side (black or white). The board would light up to indicate where the other player pieces. and the colour coded red for rooks, green for Knights, blue for bishops, purple for Queens, Pawns are white and yellow slash gold for kings. Once the person on the other side moves the pieces, the light would change on your side. It’s a really strong interaction with friends rather than just play it online. The use of the physical environment and the light colours with different representations makes them feel like the other player is right in front of them.

Our project connects people in a thoughtful way. Once a person wants to talk with their friend, there must be a reason, maybe it is what happened to them, or it may be a topic to be discussed, mood and emotion may be important reasons for chatting. Just like what happened to them that made them happy, they wanted to share with their friends. If they are angry, they need someone to complain. We can know the state of our friends in the first place and signal them back to tell us we are available. The relationship between friends or lovers would bond two people together, they could have a personalized interaction with each other. They could decide what each of the 9 LEDs represent, to make it their own password. 

As we decided each LEDs represent a mood, we used two different ways to know what we want to tell. Jay is using the labels, he sticks them next to the light to know which mood he wants to message to Bernice. Bernice is using the colours of paper and lights to know the mood. We want to make the work more decorative, so Bernice folds the origami lamps on each LED with various colour paper. Red represents angry, blue is joy, white is fresh, yellow is happy, pink is curious, green is hungry, black is bored, purple is fear and orange is excited. The colourful papercraft looks like the wagashi dessert on the plate. 

In our project, each touch button can instantly blink the LED, and the LED on the other users’ side would remain on until the other user turns it off. This is an exchange of our state to each other. NINE used 9 lights to tell us how your friend is doing. Sometimes, to know how a friend feels, the object could be more representative than the words. The combination of remote presence and physical touch makes people feel closer rather than video chatting or texting, it’s closer to in-person interaction.




Sarah Al-Mutlaq, Sarah. “T³: Connected Chess Boards.” News – SparkFun Electronics,


Uncommon Goods Blog Team. “Touch, Technology, and Giving Back: Meet the Creators of the Long Distance Friendship Lamp -.” The Goods, 24 July. 2017,


Happy Accident


Project Description:

“Happy accidents” is a series of interactive experiments that making the viewers create their own artworks. They are using their body as controllers. Every single movement could create a random art piece. Every viewer could be a creator.

They can close their eyes, move their head, raise or twist their arms to form a picture. They are using the brush with random in size, colour, and transparency. I want my audiences to be part of my work, to know every movement could be meaningful and unique.

When I was painting in the studio, I saw many painters used the pouring medium or splattering paints to form an abstract painting. Those works are composed of random accidents, that every movement of the painter could cause totally different results. Those painting techniques are not difficult for artists at all levels. You could form a beautiful painting without advanced skills. So that I want to make it works in digital arts too, to use our own movement to create a random piece of art.

I observed how to replace the mouse clicking with the interaction of computer visions. The body controller creates a more happy accident as it captured more movement other than only clicking with one finger. I want to create a physical and visual interaction experience for the viewers, to make them enjoy playing and creating at the same time.

I used PoseNet’s body tracking for all of my sketches, I tried to make a more consistent style for the artworks, with a different parts of the body to control.

Sketch One: Horoscope

I used the symmetry technique in p5js, to make pattern symmetry twice. Symmetry is an excellent technique for abstract paintings; this could turn a random shape or line into a beautiful art piece. In this sketch, I made the stroke dot-like and black background. This looks like art about the galaxy. The stroke will track our nose, and the viewers could close their eyes, move their heads, create their own horoscope, to build their universe. The speed of movement could influence the space between each dot. The viewers could also leave the page open, do other stuff on the computer, then come back to see how it turns out. It captures every small movement from our nose. Everyone could have a unique image they made by themselves; those artworks could represent their mind; they might feel related to the image; they are the author themselves.






Sketch Two: Nose Random Brush

As I am using the randomness to create abstract art, I am literally using random techniques in P5js. In this sketch, I kept using nose tracking and present. The viewers are controlling the circle brush with their nose, to paint the canvas. The circle is in random transparency, different color tone, and size. The space between each circle depends on the speed you move your nose.





Sketch 2.5: Nose Random Changing Colour Brush

I want to play with more color and randomness, so there would be a rectangular canvas. The stroke is changing the transparency and color in a different area of the canvas. This is quite similar to sketch two, so I count them as one.





Sketch Three: Black Pendulum Brush

Without the Postnet, it is a sketch that makes the center of the canvas link with every spot we pressed, now it is nose tracking the spot. With the line of it, it could from a wave or a sketchy flower, I add the random circle on the nose track. It looks like the pendulum waves. The lines could overlap on each other, or it would also keep overlapping if your nose stays in the same spot. It created the shades with the movement.





%e4%b8%8b%e8%bd%bd-2 %e4%b8%8b%e8%bd%bd%e7%9a%84%e5%89%af%e6%9c%ac %e4%b8%8b%e8%bd%bd-1%e7%9a%84%e5%89%af%e6%9c%ac

Sketch Four: Coloured Pendulum Brush

With the black pendulum Brush, I also want to try it with transparency and color. The pendulum waves reminded me of circular weaving. Other than symmetry, circular arts are also a beautiful type of art. I decrease the Stroke weight, to make the viewers focus more on the circles. It is like a digital weaving.





Sketch 4.5: Create with Wrist

Playing with more colors, I want to make track of something other than the nose. I decide to use wrists, as we have two wrists, I want to create two color tones. Because the color tracking our wrists, the colors are not divided into left and right; we could cross our wrists to make the colors cross over.





Sketch Five: Wrist and Elbow Brush

In the last sketch, I want to include the elbows, so we have four controllers and four color tones. As the wrist and elbow are quite close, so this time I don’t use the center to link the elbow, I used bottom. The lines separate the colored circle from wrists and elbows. The stoke weight is very small, it looks like blur dots.





Project Context

The context for these sketches are inspired by abstract artist, Jackson Pollock and the interactive artist, Daniel Rozin.

In the inkblot art, they are using the randomness to paint, the artwork is “predictably unpredictable, it’s ebb and flow are accompanied by such ephemeral byproducts as the inkblot and blotting paper.”(James) It is a bilateral symmetry pattern, they “catalysts for the imagination and triggers for free association.”(James) Everyone could have a unique image they made by themselves; those artworks could represent their mind; they might feel related to the image; they are the author themselves. They are creating their own Rorschach. They could use their creativity to think deep into their images. Those could help their artistic motivation.

Jackson Pollock is famous for abstract expressionism. He swapped his paintbrush, “through the dripping method, the latter fidgeted frantically to create a perfect combination of colors and texture.”(Jones) Most of Pollock’s paintings are paint splatter,  we could see how he waved the colors onto the canvas. There are many movements in both his painting and the way he paints. I want to use that in digital paintings. Splattering paints might not work with the digital panels, but it could work with computer vision. The speed, the strength, the amount of the movements could influence the result of our final work. Every single small movement could cause a different result in the painting. Those could be randomness and happy accidents, but with the creativities.

Daniel Rozin is famous for his mechanical mirrors. He used the non-reflective materials to reflect the viewers’ silhouette. He used the camera to catch the viewers and reflect them with his mirror. His works are interactive, it would not move or look anything special without the interaction from the viewers. Rozin mentioned: “My pieces are very boring when there’s not a person in front of them.”  Same as mine, my works are blank without audiences. I want to make the audience to find themselves in my sketches, they are my content. Rozin’s artworks are awesome, but he makes the audience a part of his artwork.

I want to make the viewers find themselves, everyone could make be an author to create beautiful art. This is a way to make the viewers know they are meaningful, a way to make artists help viewers.


Latta, Richard. “Inkblot Art Images.” Design, vol. 74, no. 5, 1973, pp. 2. ProQuest,

James, Emily. “The Modernist Inkblot.” Twentieth-Century Literature, vol. 63 no. 3, 2017, p. 299-328. Project MUSE

Jones, Vinciane. “10 Things to Know about Jackson Pollock.” Artsper Magazine, 20 Feb. 2020,

Staff, WIRED. “This Artist Makes Kinetic ‘Mirrors’ That Echo Your Movements.” Wired, Conde Nast, 18 June 2019,