Physical Interface w Digital Feedback: MIDI Data using Unity/Ableton/MaxMSP with Soft Tactile Buttons [Analog and Digital, Cap Sens

created by Denzel Arthur & Angela Zhang

For this final assignment, we decided to continue working on what we explored for the previous project, which was controlling a 3d object in a game engine or some other digital environment with non orthodox physical input switches. We made a lot of good progress during the initial experiment, and because we have an affinity for music, we decided to continue pursuing it for this next project, to solidify a foundation for how we create multimedia works in the future. The initial goal was to find a unique way of visualizing audio, and having it be interactive, either in the progression of the music itself, sonic feedback, or changes in the visual component that corresponds to the music, visual feedback. This lead us to experimenting with the Unity game engine, Ableton, and building physical buttons for extended and an unusual combination of hard and soft user inputs.

img_7724
conductive thread

 

 

arduino micro
arduino micro
conductive ribbon
conductive ribbon

We wanted a variety of tactile experiences on our game-board, including both analog and digital inputs and outputs. Using methods explored in class, we used materials such as conductive thread, conductive fabric, velostat, and some insulating materials such as felt and foam to interface the buttons and the electronic elements. We also wanted to use some elements that would normally be used in regular electronic setups but not necessarily in a soft circuit, such as light cells, infrared sensors, and actual tactile buttons.

schematic for digital button
Angela’s schematic for digital button, [+] and [-] intertwined but not touching, in order to maximize the surface area that can be activated.
schematic for analogue button
Angela’s schematic for analogue button [top down view]
diagram of construction of velostat button
Angela’s schematic diagram: construction of velostat button [layers, component view]

mid production - analogue button, velostat, conductive thread, conductive fabric, felt, foam, embroidery floss.

Our first soft button button was an analogue pressure sensing button made of coelostat between two pieces of conductive fabric, with 3 connecting lines of conductive thread sewn into each of the pieces of conductive fabric on either side of the velostat, in the middle. One of the sides of conductive thread is positive, the other negative. These are sewn to the edge of the square that is cut, deemed to be the button, and come off of the square approx 2 cm away from each other, and are eventually sewn into the blue felt that becomes the base of the button. The yellow foam and red felt are added for haptic feedback of a range of pressure, and the idea was to hopefully allow for a wider range of pressure sensitivity from the velostat, as well as aesthetic purpose. The without the added layers of material the button felt very flat and there did not seem to be a big margin of input for the user, especially as an analog input which is meant to provide a range of numerical data, which would then be used to control some element in Unity, as for the other components of the project. 

The completed pressure button on the board, and a light cell beside it. Two bits of silver thread on the button are conductive; one is positive and one is negative, detecting how much the velostat is being pressed.
legs of the components are fed through the board, and twisted to be flush to the back of board so they can be sewn into the prototyping board, also on the back.

a tactile button, three 5mm neopixel LEDs, and the analog pressure button, with some conductive thread connections [front view]
LED connections sewn with conductive thread to the prototyping board [mid prod, back view]
LED connections sewn with conductive thread to the prototyping board [mid prod, back view]

The main idea was to use these buttons to control gameplay within Unity for a game that Denzel had programmed for Experiment 3. The Arduino Micro however, as well as the Touch Board by Bare Conductive that Angela used for Experiment 3 to create the conductive painting [to be used as input for Unity as well, but was last used for Ableton] both have the ability to send MIDI data. We decided to switch it up and see if we could get a 3D object from Unity to work with Ableton and Max MSP’s patch for Ableton, Max for Live, to make it respond in real time to a MIDI signal sent from one of our buttons or sensors. Unfortunately we did not have time to hook up all the different sensors and components, but there is potential to hook up all sorts of different sensors and to keep adding different analog and digital parameters to the board and this is going to be an ongoing experiment for us to see how many different components we can combine.

For the connection of Unity -> Ableton -> Max:

screenshot: GitHub for The Conductor
screenshot-309
screenshot – YouTube video describing connection of Unity and Max using The Conductor
screenshot – Ableton 10 Remote Script OSC to MIDI Note forum
screenshot – we looked at some of the possibilities between Ableton and Max using Max for Live
screenshot-315
mapping audio parameters (OSC data) from ableton to Max to visualize change in audio through color mapping

The final product is a set of buttons that sends a MIDI signal from the button, to Ableton, which triggers or changes a sound. The track that the sound is on has a script that connects its phenomena to Max, which takes the MIDI data it receives and outputs a corresponding behaviour in the 3d object in either the change of its shape or colours, relative to the change happening in the sonic component. In theory anything that sends a MIDI signal can work with this setup, so it can work with both soft circuit buttons, conductive paint, or any regular MIDI keyboard; any input device works as long as you can get it to communicate with MIDI data. We experimented with other MIDIs such as the OP-1 [by Teenage Engineering] as well as the conductive painting [uses Touch Board by Bare Conductive] from the previous experiment, which outputs MIDI.

img_4594
Works with Touch Board – which outputs MIDI

56520213087__f2ad6f2b-0035-49e4-b108-a5b923533654

The Conductor – Initial setup inside Unity
Ableton - Live view; jit.window Max object with Ableton tracks (audio source)
Final set up – jit.window Max object, Max patches and MIDI tracks in Ableton Live View (audio source)

 

Yay!! Now you can be a VJ 😀

Leave a Reply