Experiment 3: Toil & Trouble


1.0 Requirements
The requirement for this project was the inclusion of a tangible or tactile interface for a screen based interaction, with a strong conceptual and aesthetic relationship between the physical and the events happening on the screen.

2.0 Planning & Context

For this I looked into exploring three input methods of physical interaction that would affect an onscreen display generated using Processing.js:
– A push button highly integrated into the design and structure of the piece
– Tactile switch using copper wire to open and close a circuit
– Magnetism and Hall Effect Sensors to return a digital signal when in the presence of magnetism
– Elapsed time within a state

For the screen output in Processing.js, I demonstrated:
– Animation – using sprites & redrawn still images
– Video
– Sound
– Images
– Text
– Drawing Simple Shapes

The presentation of the Processing output is done via a link to an iPad on the front of the piece, ideally, hiding away the interface with the computer. In the initial testing, an iPad Pro was used and SideCar was utilised to make the connection to Processing seem magical. Unfortunately, the iPad pros in the AV Room have been out of commission for the past few weeks, so the tethering had to be done via a USB and Duet Display. Wireless Duet Display connectivity was possible, but tested to be unreliable. The computer was hidden under the table.

This project was built during Halloween, and featured the process of casting a spell to create a dragon (my Chinese Zodiac and favourite mythical creature). The altar is switched on, a grimoire opened, and ingredients are added one by one to the cauldron, and after the last item is added, the cauldron boils furiously and settles. The spell is then cast over the cauldron using a hex bag, and a dragon emerges. Each physical movement by the caster triggers a switch.

It is meant to be a playful piece, performed by a single person to an audience as entertainment. The process can then be repeated by observers  interested in how it works, and what the switches are, or with the animations on screen.

This project plays with hidden switches, the aim to make the magic seem magically and not reveal exactly where the switches are or how they work. There is a bit of mystery behind what is triggering the changes, and few actual buttons are pressed or manipulated.

I initially made prototypes for the use of velostat, light sensors, electromagnets, reed switches, and servos, but found the velostat to unreliable for the purposes I needed it for, electromagnets to be too weak to lift items, reed switches to be too fragile when manipulating, lights sensors to be fickle (even when averaging the current light upon each start up), and the servos to not be powerful enough to achieve the desired effect I wanted. (The iterations can be found in the code.)

Planning Details
> Sensors and actuators
(Digital) Push button -> push down button. turns on heat. turns on array of orange lights underneath, and turn on/off levitation
(Digital) Push button -> ingredients reset button
(Analog) Tactile button x 4 -> ingredients complete circuit. tap a few times before removing.
(Digital) Magnetic Switch -> hex bag with magnets to trigger incantation.
(Digital) Lights -> triggered at various stages

> Casting modes
-1 Pre-activation
0 (Activated)
1 (Ingredient 1)
2 (Ingredient 2)
3 (Ingredient 3)
4 (Ingredient 4)
5 (Stew Ready)
6 (Spell)
7 (Dragon)
8 (Off)

Note: Although two Arduinos are pictured, only one was used. The other was kept in to allow for rapid switching in the event of failure, which it did shortly before presentation. Ultimately the Mega was used as the “Uno” (actually a knockoff Arduino) began to fail unexpectedly. I found out after that it not being a genuine Arduino was the cause of the intermittent failures in this experiment, and also in previous experiments where we relied on it. It was too late for other experiments, but just in time for this one.

3.0 Implementation


Cauldron with ingredients, on stove and “wood” (actually small strips of acrylic).

3.1 Software & Elements

3.1.1 Libraries & Code Design

Code: https://github.com/jevi-me/CC19-EXP-3

As this was a solo experiment, I took the opportunity to explore the software design capabilities within Arduino. I wrote and adapted various classes for use within the code, and tried coding in a more object oriented way, just for experimental purposes. I started off with quite a number of borrowed and written classes and libraries (over 10 at one point), but slowly as features were cut, repurposed or shifted, the number of imported libraries was lowered to 3. The experience was very fulfilling however, and exposed me to a different way of communication with the device.

Arduino Loop Method:
A) Wait for Activation button to be triggered. If the stove isn’t on, nothing works.
– turns on stove lights, turns on altar lights
– change bool
– set state
– send state
B) Check for a change in pressure on the ingredient bases (& if activated)
– change lights
– change bool
– set state
– send state.
C) If all ingredients are added, stew is ready after boiling for a few seconds
– change lights
– change bool
– set state
– send state
D) Hall switch triggered (& if stewready)
– change lights
– change bool
– set state
– send state
X) Sync the info with Processing periodically

On the processing side, I was eager to explore as many of the output features that were available. I developed methods that would allow for animation using custom-made and premade sprites or still images running at different frame rates and loop/no-loop variables, videos at different locations and triggered to start based on the input from Arduino, and sound that started, looped or not, volume changed, and stopped based on the state of the system. I also used text, and the drawing functions of Processing. As in the case with Arduino, the aim was to explore as much of the capabilities that Processing.js had to offer.

In both Arduino and Processing, I explored the use of fail-safe methods that allowed the intertwined systems to fail gracefully and return helpful feedback to the user. Though not a requirement of this project, it was again, an exploration of the capabilities. Examples of this is if the state of the Serial Port, and another the status of USB tether between Arduino and Processing. In both cases, the system fails gracefully, and provides help hints on screen and/or in the console to help guide the method of correction.

3.1.2 Sound Files
Various sounds were used throughout:
– Background music played before the grimoire is opened
– Cauldron boiling sound — there are two types depending on the reaction in the cauldron taking place, and they increase with intensity as the ingredients are added
– “Double Double Toil and Trouble” from Shakespeare is read during the spell casting
– Background music played when the dragon is successfully created

3.1.3 Videos
Four videos are used in this project. The videos were placed on different locations of the screen when triggered to act as design flourishes. The videos were sourced from an Instagram video pack, scaled, cleaned up, and formatted appropriated for use within Processing. The videos serve as the lowest layer of the elements on the screen.

3.1.4 Animation – Sprites
The bunny animation on the lower right corner reacts to the state that the cauldron is in — each new action performed by the caster, or in some cases, a timed reaction, changes the animation of the bunny. It can sleep, run, sit, rise up or lower itself at different levels in response to the caster. The animation uses carefully named and numbered sprites and a customised method to cycle through the sprites drawing them to the screen, looping around to the beginning of the list if the animation cycles within the state. Since the framerate that Processing draws at is fixed, to control the framerate of the animations and save on the number of sprites used, I wrote a frame skipper, that allows for a number of Processing frames to be skipped over and not have the animation redrawn.

The cauldron sprite images were made by taking screenshots of a 3D image of three similarly designed cauldrons, as I rotated them around at different angles. As with the bunny animation, they were carefully named and numbered to allow the cauldron to spin, and different versions of the cauldron were drawn based on the stage of the spell.

3.1.5 Animation – Still Images
The grimoire can be open or closed, and is animated to mimic levitation using simple oscillating redrawing on the y-axis.

The dragon utilises the same “oscillating on the y-axis” to mimic flight, and only appears if all the previous steps have been completed. This information is received from Arduino which controls the logic.

3.1.6 Images
Representations of the physical images appear on screen when they are added to the cauldron, and are listed all together at the beginning. The position, and visibility of the images are tied to the physical interaction taking place.

3.1.7 Text and Drawing
I used text in various places to indicate the state of the spell and explain what is happening. Thre is also a small indicated on the bottom left that shows what stage the spell is in.

3.2 Hardware
Copper tape and alligator clips make the main tactile interaction. The circuit is completed when the ingredient in the storage basin of the piece is in its place. On its removal, the switch is “released”, as it the circuit is opened. This open circuit is what triggers actions processed by Arduino and sent to Processing.

Another switch used is the detection of a magnetic field. A circle of “diamonds” marks the spot where a hex bag is to be placed. The hex bag contains 4 neodymium magnets which create a magnetic field that is registered by a hall detector in place just under the wood.

The other types of switches are ordinary push buttons which were soldered and positioned flush to the surface of the piece allowing them to blend into the design, and time based “switches” which trigger after a predetermined time has elapsed e.g. during boiling and during incantations.

Other things of note in the build are the ingredients used, some real and natural ingredients like tulip bulbs and spices, and others symbolic like skeletons and eyes. Ingredients list: eyes, pumpkin, magic, winged bats, tulip bulb, mushrooms, skeleton of a snake (the word ‘dragons’ come from ‘draco’ and ‘drakon’ meaning ‘large serpent’ or ‘sea serpent’), orb, candle. Many These were purchased at Dollarama during the Halloween season.

Lights were used as a reference of what was happening on screen, mimicking a magical altar, and a flame below the stove under the cauldron.

The altar was laser cut and featured steampunk gears and a skeleton cup which held the “dried blood of lizard”. The stove also laser cut, used wall screws and brackets to add height, and had acrylic strips to mimic wood in the fire pit.

4.0 Reflections

The piece is meant to be a performance using sleight of hand and hidden switches, but it also encourages others to want to try it, and make a dragon as well. This then requires to reveal the switches, and it isn’t very obvious how things work. Another iteration of this could be a more DIY version where there is better labelling. That would make the system for others rather than a performance piece like this was created. In testing with others previously, I received the comment that it could be cool if different ingredients and patterns yielded different creatures. That would require a more complex code, but is possible as yet another iteration of this project.

5.0 Additional Photos

6.0 Literature References (MLA Cited)
Shakespeare, William, and A. R. Braunmuller. Macbeth. Cambridge University Press, 1997.
Double, Double Toil and Trouble – Shakespeare, http://www.potw.org/archive/potw283.html.
“DRAKON AITHIOPIKOS.” ETHIOPIAN DRAGON (Drakon Aithiopikos) – Giant Serpent of Greek & Roman Legend, https://www.theoi.com/Thaumasios/DrakonesAithiopikoi.html.

6.1 Additional Reference Links (Linked)
Envato Elements (elements.envato.com) – for some of the sounds, videos, and graphics.
Processing.js Guide (processingjs.org) – for part of animation code
Circuito.io (circuit.io) – for libraries. Licenses included in the source code.

Leave a Reply

Your email address will not be published. Required fields are marked *