Project Title: Live VJ Performance Show
-An interactive live performance that explored audio visualization
By Jun Li – Individual Project
This project was a live performance show based on the concept of ‘Audio Visualization’, duration of 11 mins-long show for the audience. Also, it’s my first attempt at being a VJ. All 8 of the different effects were generated and interacted with music input in real-time. It was built on the Arduino controllers, TouchDesigner in addition to projection of the video output on the background. The purpose of this experiment was to create a very simple user-interface with different switches and sliders to manipulate the effects.
This dynamic experience allows the every participant to have an opportunity to become a ‘Visual Jockey’ as, they can operate and furthermore, change each parameter of the audio resulting in creation of the energetic graphics in the background.
Keywords: VJ, Live Performance Show, Audio Visualization, Interaction.
The goal of this experiment was to create a tangible or tactile interface for a screen-based interaction that uses Arduino and Processing. Since, l came from a very similar undergraduate program called ‘Technoetic Art’ and l had a lot of experience working with these software beforehand. l am always eager to challenge myself for a high-tech level and create various fascinating projects. After discussing my idea with Kate and Nick, I received their permission. I used the technical logic of Processing and applied it to TouchDesigner. Thus, retaining the knowledge of serial communication in these 2 softwares.
Music visualization, refers to a popular way of communication that combines audio visual and audio with the core of vision, music as the carrier and various media technologies such as new media technologies to interpret music content through pictures and images. It provides an intuitive visual presentation technique for understanding, analyzing and comparing the expressiveness of internal and external structures of musical art forms.
Vision and hearing are the most important channels for human beings to perceive the outside world. They are the most natural and most common behaviors of human beings and are irreplaceable for the activities of the human cognitive world. “Watching” and “listening” are the most natural, direct, and most important means of recognizing the outside world through a variety of audiovisual senses. Sound and video, hearing and vision, in contemporary society, the two agree on aesthetic trends and dominate the aesthetic form of mass culture. Vision provides many conveniences for people to see and understand musical works and music culture. People will increasingly rely on visual forms to understand audio content. The specific application of music visualization is very wide. For example, live music, exhibition site and other music visualization systems, combined with special images and customized immersive content development works, can give people a strong visual impact while enjoying music.
Process & Techniques
First, l have explored and researched what kind of parameters from audio can affect the visual part and how can l utilize these to manipulate the shape or the transformation of the visual design. Usually, audio visualization is based on the high, mid, low frequency, beat and volume of the music. So, one of the most important techniques is how to get these data or parameters from the audio itself and how to convert it to audio visualization.
Most of my project program was built in TouchDesigner, for the Arduino part, the code was not difficult, just sent the data of a switch and 8 sliders to TD for manipulating the visual effect through serial communication.
- How to generate visual effects in real-time based on the different parameters of audio?
- How to utilize the data coming from Arduino to affect the visual effects that already had been generated in real-time?
- How to arrange and switch different visual effects and what’s the key and special point for VJ performance.
- How to optimize the program to avoid affecting the real-time output.
- How to output the whole project (both visual and sound part) to large displays (2 screen monitors, 1 projection) and 2 speakers to let VJ performer monitor and change the effects in real-time and show it to the audiences.
For the first and second questions, l had created 8 effects, utilizing different parameters from audio.
1.1 – Audio analyzing (high, mid, low frequency)
1.2 – Utilize the data from Arduino to manipulate visual effects
1.3 – Opening introduction (particle writing system)
1.4 – 8 different visual effects (Initial state & Altered state, Same below)
For the third question, I used the data from the slider and added an additional layer in black to switching and transited to the next effect. At the same time, l explored a lot of VJ will add strobe visual effect while the music reached its climax. So, there was another additional layer based on the low frequency of the audio, it increased the strobe of the video, which could improve and enhance the live atmosphere.
1.5 – The interface of switching 8 effects
1.6 – Different function in layers
For the fourth question, in the beginning, functionally l could utilize one data from the slider to control all 8 visual effects. However, l realized it was a huge challenge for my laptop to process it on the GPU & CPU at the same time. So, l had to separate the data into 8 sliders to control each effect and test it carefully. Eventually, it worked and did optimize my program successfully.
1.7 – The data flow in Touchdesigner
1.8 – Arudino interface
For the last question, l did explore the function in TD for project output, it was powerful to allow the creators to create a super convenience interface to supervise the whole process and outputted it to any screen and speaker.
1.9 – Output to display
Challenge & Learning
- The due time of this project was super short. It required us to submit in 9 days, including having ideas, coding, testing, debugging and final setup. Especially, l used a more difficult software – TouchDeisgner. Although l had experience working with TD before, not as familiar as Processing. At the same time, l didn’t have much project experience. l barely got any help and reference from my classmates. It’s really an individual work and quite challenging for me, which l am excited as well.
- Set a goal to a live VJ performance show means too few visual effects won’t be accepted and not powerful enough in visual and hearing. I am a perfectionist. So, I had to create 10 mins-long show at least, that’s the extra requirement and pressure for me. So, to achieve that in the end, l kept testing and creating different visual effects. Eventually, l chose 8 powerfully and great enough from 26 effects l created in TD. 2.1 – Reference library l created
- Processing data and having these effects in real-time were a super heavy job on my GPU and CPU. it also challenged the performance of my computer. l almost getting closer to ruin my GPU and unfortunately, l still lost my 3.5mm audio output in the end. After coding, l had to work really hard to optimize the program to get a better, stable 60 FPS output. Because my computer was not powerful enough, especially the GPU. Eventually，there still were some frame droppings during live performances.
- To my satisfaction, this project was successful been made and the response from others were very great. l perfectly achieved the goal l set in the beginning, compared the initial goal, NOTHING had been changed. I learnt a lot of project experience and techniques in TD, including analyzing audio, visual designing, data integration, manipulation and analysis, project optimization, output and management.
- The next step l will do is to keep improving and optimizing my program and create a simpler operation interface, which can let users manipulate it easily.
2.2- VJ setup in TouchDesigner
Currently, we use technology-based new media art as a way out. These forms of new media art are often the forerunners of future art. Artists work in the area of “art and technology” to create collaborative artworks. What’s more, in the era of such a big technological transition in modern times, the way of life has changed accordingly, and it has become necessary to fundamentally grasp the essence of art. The cross-border nature of music visualization art is very obvious. It involves music art, visual art and the artistic integration of the two.
When this project was exhibited for the exhibition, I was the VJ and presented it myself and able to personally observe the interaction and experience of the participants with it. To my satisfaction，I achieved my goal of becoming a VJ, bringing a show which challenging me incredibly.
- vjfader (2017). VJ Set – Sikdope live @ Mad House – ShenZhen China.Available at: https://www.youtube.com/watch?v=uG1GrD7VQOs&t=902s.
- Transmission Festival (2017). Armin van Buuren & Vini Vici ft. Hilight Tribe – Great Spirit (Live at Transmission Prague 2016). Available at: https://www.youtube.com/watch?v=0ohuSUNHePA
- Ragan, M. (2015). THP 494 & 598 | Simple Live Set Up | TouchDesigner.Available at: https://www.youtube.com/watch?v=O-CyWhN4ivI
- Ragan, M. (2015). THP 494 & 598 | Large Display Arrangement | TouchDesigner. Available at: https://www.youtube.com/watch?time_continue=1&v=RVqNjJfE9Lg
- Ragan, M. (2015). Advanced Instancing | Pixel Mapping Geometry | TouchDesigner. Available at: https://matthewragan.com/2015/08/18/advanced-instancing-pixel-mapping-geometry-touchdesigner/
- Ragan, M. (2013). The Feedback TOP | TouchDesigner. Available at: https://matthewragan.com/2013/06/16/the-feedback-top-touchdesigner/