Experiment 1 – Visualization of Gossip | Siyu Sun

—— Gossip can destroy a person or make them strong.





Visualization of Gossip(2021) is an immersive interactive narrative work between the human and the environment.

The technique I used here is based on the case study in ml5, PoseNet, Sound Visualization in p5.js. So audiences don’t have to use the external input device, such as use mouse to implement click events. I set up creatCapture() in order to connect the camera, then combined it with PoseNet’s tracking system, and “scroll”, “click” experiment, to get feedback on the sound, graphics, and status of the control screen.

Back to the creative part, this work is divided into four narrative clues. The control feedback system and purpose of each theme are different. In combination with the research done in Experiment 1, I hope to establish an immersive area combining audiovisual and express my conceptual understanding of “Rumors/Gossip”.  I used this technique of expression due to I am concerned that the media has the potential to attract multiple senses through rich information, and has plenty of potentials to influence audiences in perception, cognition, and emotion. The sensory or perceptual mode, surround effect, and resolution in the immersive experience will help the audience create a sense of presence in virtual events, and associate the sense of consciousness or substitute reality. [1]





Study One: Movement and Feedback | ml5, createCapture(), preload()

The prototype of Gossip

This is my first work. I use harsh noise images to show what external evaluation brings. The cacophony of music goes up and down. When people hold their head and rock from side to side, the noise image follows people’s movement, it’s like noise in your head that you can’t get rid of, swirling back and forth.  

The most technique I study in this part is knowing the ml5,PostNet, setting up a Creaturecapture function, and know-how to preload media files.  The most key thing I would like to mention is I used the person’s silhouette instead of a real person over the camera, that’s because I want the whole works to have a sense of unity. So in order to  realize it, the important thing you must do is upload media file in function preload(){

Then, I combined the Capture function of ML5. PoseNet to Capture the position of the audience’s Nose to control the movement of the background picture, and set the noise image at the position of the eyes and Nose to adjust the specific orientation and make it follow the movement of the audience so that it is difficult to get rid of this concept.  

How to do 

Hold your head and shake left and right.


The problem I met in this process, the visual identification is not particularly accurate and it’s better to change the soundwave image to true vibration frequency, but I don’t know how to combine them together.


createCapture():Creates a new HTML5 <video> element that contains the audio/video feed from a webcam. The element is separate from the canvas and is displayed by default. The element can be hidden using.

Preload():  This function is used to handle the asynchronous loading of external files in a blocking way.


p5.js Present:   https://preview.p5js.org/lizsun703/present/Qhbzo9zdl

p5.js Edit:         https://editor.p5js.org/lizsun703/sketches/Qhbzo9zdl



Study Two: Movement As a Click | MousePressed()

— Don’t be afraid

When gossip invades your brain, the only thing you have to do now is beat them and rebel against the odds.  In this part, I combined the Click function of mousePressed to change the image.  I’ve set up two different figures — controlling the variation of the picture by identifying the viewer’s waist height, and the noisy image disappearing when the viewer raises their hand.  It stands for defeating the gossip.  

How to do

Move from side to side and raise your hands to see the image disappear.


However, the biggest regret in this part is that I want to make the noise sound disappear when I raise my hand, but I have tried many times without success. I hope to continue to study this question in the next few days.


MousePressed()This function is called once after every time a mouse button is pressed over the element. Some mobile browsers may also trigger this event on a touch screen if the user performs a quick tap. This can be used to attach element-specific event listeners.



p5.js Present: https://preview.p5js.org/lizsun703/present/M38Co7k5r

p5.js Edit:       https://editor.p5js.org/lizsun703/sketches/M38Co7k5r



Study Three: Sound Visualization | waveform()

— Self-digestion

This is an area full of noise and language.  The inspiration of this work comes from the interactive text experience of “IN ORDER TO CONTROL” In this study, sound visualized learning was conducted to determine the waveform to read the amplitude of sound and draw the waveform of the sound. The mic was also set so that the audience could speak and the amplitude would also be realized.  And I set up a lot of words on the screen, praise, criticism, insult, people can speak any works in here and can see the different amplitude was changed. It’s really funny that it’s my first time to try this part, and I saw the tutorial to wrote down the code in random(in order to try), then I got the effect I want to have haha—- The soundwave was a fusion with the profile, that means people can make balance with gossip.

How to do

In this process you need to watch the text on the screen and listen to the soundwave noise, you can speak anything you want, and the microphone will record the voice then feedback to the waveform.


In the beginning, I designed to make these sentences scroll up and down, but I tried it hard to be scrolling them together. 




p5.js Present: https://preview.p5js.org/lizsun703/present/BsiRBoJzX

p5.js Edit:       https://editor.p5js.org/lizsun703/sketches/BsiRBoJzX



Study Four: Dynamic Text

— Balance yourself

After experiencing the whole work, I hope to let the audience know how to establish their own balance point in the gossip: We live in a space full of all kinds of external voices, and all kinds of gossip, like knives and stones, affect us, bring us to pressure, but also label us. It made me imagine the weight of pressure, rumors can destroy a person or make them powerful.

How to do

In this process, you need to raise your hand and move your body everywhere you want. I  set up a bad sentence ” You’re a jerk!” on the left, and the praise sentence”You’re so creative!” on the right. You can find it to be in a balanced way.


p5.js Present: https://preview.p5js.org/lizsun703/present/8XF0zEzkO

p5.js Edit:       https://editor.p5js.org/lizsun703/sketches/8XF0zEzkO




As I mentioned before, the sound visual study is my favorite part and I do the related works since 2019, using Kinect combined TouchDesigner.

Siyu Sun: ” RUMOR” Kinect with TouchDesigner interactive art device (vimeo.com)



[1]Teresa Chambel. 2016.  Interactive and Immersive Media Experiences.  In Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web (Webmedia ’16).  Association for Computing Machinery, New York, NY, USA, 1.  DOI:https://doi-org.ocadu.idm.oclc.org/10.1145/2976796.2984746