created by Denzel Arthur & Angela Zhang
For this final assignment, we decided to continue working on what we explored for the previous project, which was controlling a 3d object in a game engine or some other digital environment with non orthodox physical input switches. We made a lot of good progress during the initial experiment, and because we have an affinity for music, we decided to continue pursuing it for this next project, to solidify a foundation for how we create multimedia works in the future. The initial goal was to find a unique way of visualizing audio, and having it be interactive, either in the progression of the music itself, sonic feedback, or changes in the visual component that corresponds to the music, visual feedback. This lead us to experimenting with the Unity game engine, Ableton, and building physical buttons for extended and an unusual combination of hard and soft user inputs.
We wanted a variety of tactile experiences on our game-board, including both analog and digital inputs and outputs. Using methods explored in class, we used materials such as conductive thread, conductive fabric, velostat, and some insulating materials such as felt and foam to interface the buttons and the electronic elements. We also wanted to use some elements that would normally be used in regular electronic setups but not necessarily in a soft circuit, such as light cells, infrared sensors, and actual tactile buttons.
Our first soft button button was an analogue pressure sensing button made of coelostat between two pieces of conductive fabric, with 3 connecting lines of conductive thread sewn into each of the pieces of conductive fabric on either side of the velostat, in the middle. One of the sides of conductive thread is positive, the other negative. These are sewn to the edge of the square that is cut, deemed to be the button, and come off of the square approx 2 cm away from each other, and are eventually sewn into the blue felt that becomes the base of the button. The yellow foam and red felt are added for haptic feedback of a range of pressure, and the idea was to hopefully allow for a wider range of pressure sensitivity from the velostat, as well as aesthetic purpose. The without the added layers of material the button felt very flat and there did not seem to be a big margin of input for the user, especially as an analog input which is meant to provide a range of numerical data, which would then be used to control some element in Unity, as for the other components of the project.
The main idea was to use these buttons to control gameplay within Unity for a game that Denzel had programmed for Experiment 3. The Arduino Micro however, as well as the Touch Board by Bare Conductive that Angela used for Experiment 3 to create the conductive painting [to be used as input for Unity as well, but was last used for Ableton] both have the ability to send MIDI data. We decided to switch it up and see if we could get a 3D object from Unity to work with Ableton and Max MSP’s patch for Ableton, Max for Live, to make it respond in real time to a MIDI signal sent from one of our buttons or sensors. Unfortunately we did not have time to hook up all the different sensors and components, but there is potential to hook up all sorts of different sensors and to keep adding different analog and digital parameters to the board and this is going to be an ongoing experiment for us to see how many different components we can combine.
For the connection of Unity -> Ableton -> Max:
The final product is a set of buttons that sends a MIDI signal from the button, to Ableton, which triggers or changes a sound. The track that the sound is on has a script that connects its phenomena to Max, which takes the MIDI data it receives and outputs a corresponding behaviour in the 3d object in either the change of its shape or colours, relative to the change happening in the sonic component. In theory anything that sends a MIDI signal can work with this setup, so it can work with both soft circuit buttons, conductive paint, or any regular MIDI keyboard; any input device works as long as you can get it to communicate with MIDI data. We experimented with other MIDIs such as the OP-1 [by Teenage Engineering] as well as the conductive painting [uses Touch Board by Bare Conductive] from the previous experiment, which outputs MIDI.
Yay!! Now you can be a VJ 😀