Hey P5.js! Change colours with my movement

I developed this idea because I was very interested in how body movements can interact with technology to connect with shapes and colours. I always have had a fascination with colours and how they affect the mind and body. For example, in my apartment, almost every light bulb has a colour mode which I can change with a simple voice command. I am interested in being able to explore how I can move my body and have the lights change according to my movements. Therefore, I put those ideas to use with this assignment.


In these projects, I innovated with four interactive studies using four different movements to change colour and move shapes. I used P5.js with Posenet, which is a body-tracking program that detects different body joints to form a skeleton, and estimates a human pose.  Users can move different body parts to interact with this program to explore various shapes and colours using scroll and click functions.


I tried to make this program user-friendly and intuitive, and to ensure that the user has fun! This project was inspired by smart home design concepts using lights and smart assistants, such as Alexa and Siri, both of which are voice-activated. One of my biggest challenges, particularly for someone who hasn’t done coding before, was that the camera displays the inverse image of what is in front of it. So, if the users move to the right, the computer displays the movement to the left. My hope in the future, is that various appliances, like lights, televisions, and speakers can be manipulated by certain body movements that reflect our moods and desires.


Experiment 1: Shoulder Dance

The users can move their shoulders to the left or right, and the four vertical squares will move synchronously following the direction of the shoulders. This interaction requires a body scroll function. There is a bigger space between the middle two squares that allows the user to know how far from the camera he/she/they should be by aligning your eyes to fit in between the two middle squares. I chose pale pink as the colour, not just because it’s my favourite, but also because it’s a happy calming colour. This colour combined with the swaying of shoulders, hopefully makes the users feel like they are doing a gentle dance.


Present: https://preview.p5js.org/MerelVS/present/G-uXtdnkL

Edit: https://editor.p5js.org/MerelVS/sketches/G-uXtdnkL


Experiment 2: Nose Colours

There are four coloured circles in the four quadrants of the screen. A smaller turquoise circle is projected on the nose and acts as a mouse. When the users move the turquoise dot with their noses to one of the coloured circles, the colour of that circle is exchanged with colour from the circle that is positioned diagonally to it. All the circles in the four quadrants are pastel colours. This interaction requires a click function. Besides being amusing, this project reminds me of the childhood game called ‘connect the dots’.


Present: https://preview.p5js.org/MerelVS/present/HrlZqYUyn

Edit: https://editor.p5js.org/MerelVS/sketches/HrlZqYUyn


Experiment 3: The Slinky

With the movement of both eyes from one end of the screen to the other, the two circles representing the eyes move and change size and colour. When the user moves right, the circles become smaller and darker. When the users move to the left, the circles become larger and lighter. This interaction requires a scroll function. This project reminds me of a childhood toy I used to have – a rainbow coloured slinky. And the good news is this one doesn’t get tangled! Another sensation that the users may feel when interacting with this project is that they can paint circles on a blank canvass with their eyes.


Present: https://preview.p5js.org/MerelVS/present/D0qxUJnGH

Edit: https://editor.p5js.org/MerelVS/sketches/D0qxUJnGH


Experiment 4: A Colour Party

When either of the dots on the users’ wrists touches the opaque pink circle with “HI” written in it, a strobe light of random colours is projected on top of the user to mimic the feeling of being at a party. This interaction requires a click function. This project is for pure fun and amusement. The best way to use this project is by setting your computer down, standing one meter away so that the camera can detect the user’s wrists, and then raise your arms up and down and have a party!


Present: https://preview.p5js.org/MerelVS/present/LRD3bwHrx

Edit: https://editor.p5js.org/MerelVS/sketches/LRD3bwHrx