Ask Zoltar

Elliott Fienberg / Jay Irizawa / Tatiana Jennings / Monica Virtue
Project 3: City As Amusement
Creation & Computation
Digital Futures / OCAD University


Ask Zoltar is an outdoor interactive installation inspired by the fortune teller booth arcade type attractions. The traditional fortune teller booth is a self operating machine, which contains an automaton (a non-electronic moving machine in the form of a human or an animal, with a range of head and hand movements, sound and a simple interactive element such as a button).

The first mention of Automata (from Greek αὐτόματον, automaton, “acting of one’s own will”) could be traced to Greek mythology. The idea of mechanical human had been popular throughout the centuries and many intricate and accomplished examples were created to be used as amusement, tools, toys, religious idols, or prototypes for demonstrating basic scientific principles (Sharkey, 2007).


Case Studies:

We explored the following case studies to help ground our fortune telling project in time and space:

Helvetia Park – 2009

Location: Aarau, Bellinzona, MEN (Museum of Ethnography Neuchâtel)
Design: Anna Jones, Patrick Burnier, Raphaël von Allmen
Direction: Grégoire Mayor, Marc-Olivier Gonseth, Yann Laville


Helvetia Park is an interactive immersive exhibition at the Ethnography Museum Neuchatel (MEN) in Switzerland. The exhibit consists of 11 stalls inspired by the fairground attractions, including games, freak show, merry-go-round and a shooting gallery. The exhibit examines points of contact and friction between different understandings of culture in Switzerland today and comments on clash between elitism and popular amusement.


Helvetia Park is a playful artistic intervention, which satirizes museums and allows the visitors to upend the concept of value and preciousness of art.





Night Lights – 2010

Location: Auckland Ferry Building (Auckland, New Zealand)
Design: Yes Yes No (New York City, U.S.A.)


Source: LSN Global Insights

For this installation, Yes Yes No teamed up with the Auckland Ferry Building (Church) to create an immersive experience which encouraged participation and movement from visitors, thereby turning the city into an amusement park.

A series of bright, white panels were set up in front of the building, which helped create shadows which were then projected five stories above the participants. This amplified their bodies by seven times their actual height, turning the participant into something much larger.


Source: IDN World

The project not only explores the relationship of person to object (the building), but also person to person through the building, as people are encouraged to dance together.

This is a great example of how to turn the city into a playscape. It helps to (literally) shed light on the architecture of the neighbourhood, but the most interesting feature is that it isn’t that enticing until people get involved and interact with it. The bright panels not only make it possible to create shadows, but also help draw attention to the fact that you can participate.

Fun House – Nuit Blanche 2012

Location: Bay Street, Toronto Ontario
Design: Commissioned Feature Artist Thom Sokoloski

Fun House is an outdoor interactive installation conceived by artist Thom Sokoloski. Inspired by the camp and horror of carnivals from his youth, Sokoloski amplifies the banal into a sublime experience of pop culture through the projected images of old footage horror films, flashes of multicoloured lights, auditory screams and choreographed performances by post-apocalyptic carnies dragged through the mud. The contrasting areas of spectacle and experience create a cohesive yet jarring experience, not unlike the layered experiences one encounters in an entertainment venue high on multisensory stimuli output.


The main points to consider are taken from the experience of the event:

Anticipation: Attractors such as large entry-ways, flashing lights, sounds and human performance draw a crowd and entice them to come in. The promise of a changing experience awaits beyond the curtain.


Spectator to Participant: An immersive environment sets the tone, and elicits a response from the audience to actively engage and become a willing participant in the event.


Emotive responses: Cultural and subcultural signifiers elicit emotive responses, prompting memories in the form of pop-culture references, music and imagery. Visual cues such as fonts, colour, lighting, iconic objects, wardrobe, performance and other senses can leverage nostalgia connecting the audience in a familiar way.


Finale: The pace and choreography of the experience culminates to a grand moment, signifying the end of the experience, and a return to ‘normalcy’ (sometimes expedited through the egress experience).


Wonder Mountain’s Guardian – 2014

Location: Canada’s Wonderland (Toronto, Ontario)
Design: Triotech (Montreal, Quebec)

Wonder Mountain’s Guardian is a 4-D amusement ride that premiered at Canada’s Wonderland in May 2014. Designed by Triotech Studios of Montreal, the ride combines a traditional roller coaster with an immersive interactive component that utilizes custom-built projection technology. Triotech, a manufacturer of multisensory attractions for the amusement industry, developed the 3D animations, projections, special effects and sound specifically for the new ride.


Source: InPark Magazine

Guardian winds its way around theme park’s iconic Wonder Mountain on 325 metres of track, before travelling inside to five different layers of an “underground world.” It culminates in an interactive dragon’s lair built into the top of the mountain.

The new coaster is designed to appeal to gamers who enjoy a strong storytelling experience: the quest culminates as riders enter a dragon’s lair for an ultimate battle with Lord Ormaar. The ride is built for a group play experience, with the chance for individual scoring. Riders can test their gaming skills to compete against each other, with individuals able to land a coveted spot in the Wonder Mountain’s Guardian Hall of Fame.


Source: Vaughan Citizen

An accurate targeting system allows users to fire lasers from hand-held guns at real-time graphics displayed on the world’s longest interactive screen, at 200 metres long and four metres high.

In addition to the technology, the designers have overlaid the 3D experience with 4D elements, such as simulated wind and movement.

Disneyland – 2014

Location: Anaheim, California

Disney – the innovator of the amusement park concept (Gilmore and Pine: 57, 2007) – has introduced RFID technology to track large quantities of data to observe consumer behaviour and interaction with the programmed attractions (Business Week 2014).


The intent of the added feature to the operations infrastructure is two-fold: first, the mass patterns of large consumer data can be quantified in micro-elements to determine the efficiency of elements such as line-ups, circulation flow, optimization of point-of-purchase sales, and duration of captive audiences; and second, the analysis of data can be interpreted to forecast trends in the amusement park market to extend the consumer satisfaction beyond the event, as the analysis of such algorithmic trends could be tailored to offer a unique experience beyond the physical touch-point of the environment, into the psyche of the returning customer.

On one hand, RFID tracking maximizes the ROI by attending to customer satisfaction with the means to respond quicker to customers by anticipating their needs. Disneyland attendees are willing participants in the event process, paying for the experience seemingly made for their sole enjoyment. On the other hand, the capability of isolating individuals anytime, anywhere becomes a means of control, which for the most part is readily accepted by the park attendees-turned participants. On the other hand, information beyond the context of the park turns into personalized email notifications, promotional material, downloaded apps, post-experience retail consumption, and a deeper-rooted affinity to the intangible experience economy recognized by the psyche of consumers through the iconic status of animated figures across all media.

RFID may not have invented theme-park psychology, but it is the latest tool added to the history of feed-back loop marketing from large-scale amusement parks to small-scale retail places (iBeacon, Wired 2014), and is becoming commonly accepted as a pervasive yet quiet form of private information acquiescence in the finite space of the “public.” In a relevant but complex discussion to be unpacked another time, Rosalyn Deutsche makes a case to reclaim the public space from territorializing entities formed from markets and policies, citing architect and theorist Michael Sorkin, “…in the new ‘public’ spaces of the theme park or the shopping mall, speech itself is restricted: there are no demonstrations in Disneyland. The effort to reclaim the city is the struggle of democracy itself.” (Deutsche 283: 1996)

The Fortune-Telling experience is but a small scenario exemplifying an exchange between the citizen and her collection of private data as currency, in return for prophetic enlightenment in the medium of entertainment.


In the beginning of our creation process we explored various ideas inspired by the project’s theme, “City As Amusement.”  Living in a big city is a stressful and often grinding experience. Everyday’s mad rush along the streets to and from work doesn’t invite a playful mood or give an opportunity for an imaginative interlude. The streets fade into a blurred background and the passers by dissolve into a faceless crowd. An introduction of an unexpected interactive element can break the everyday routine and transform unremarkable mundane surroundings into a place of spontaneity and play.

The wireless communications nature of the assignment presented us with interesting opportunities. We explored a few different directions and scales, from transmitting a message to the CN Tower and changing colors on the CN tower’s night illumination system to creating a two person game on the playground.

One of the ideas was a wooden cabinet with lots of small drawers, which you can interact with  by writing a question on a piece of paper and depositing it in one of the drawers. The cabinet would respond by pushing out another drawer with a written answer in it. We liked a physical and surprising quality of the interaction and a challenge of building an object where the technology will not be apparent and the interaction would feel magical.


After a few iterations and research into street fortune telling and amusement park aesthetic we came up with Ask Zoltar – a fortune telling booth. The name is borrowed from the 1988 American fantasy comedy Big.

The final idea included an interactive kiosk embedded with sensors, lights, a small video screen, a receipt printer and a large video projection on the window above.

Ask Zoltar could be installed at any street corner. The booth is self-contained and can function independently for hours at a time, powered by a car battery. The projector could be installed on the top of a car and projected on a surface next to the booth.

Notes on the process:

We started with deciding what functions we wanted the box to perform, and then began creating a diagram with all of the box states and proposed functions. We wanted the box to be responsive to the environment and interactive, so we chose to use a proximity sensor, a pressure sensor, a small receipt printer, 2 strings of LED lights, XBee module, EL panel (with the shield and inverter) and a small bluetooth speaker.

We established 5 box states:

  1. Idle (The interface and screen are dark. The first loop, which is a black screen, is playing  and LED lights blink every 20 sec. There is a snoring sound coming out of the box. Zoltar is sleeping.)
  2. Sensing approach. (The proximity sensor detects a person standing in front of the box. The lights come alive and the hand EL panel begins to glow. The small screen switches to the second loop – sleeping Zoltar. You can see his eyes moving as he sleeps.)
  3. Hand press. (The viewer placed his/her hand on the palm print initiating the pressure sensor. The lights start blinking and the third loop starts playing. Zoltar wakes up, his eyes open and the screen displays a countdown from 10 to 0.)
  4. Projection. (The projection on the window is initiated through the XBee, while the small screen continuing to play the 3d loop, which is now displaying a slowly moving graphics).
  5. The fortune. (The projection stops and the small screen, which is still playing the 3rd loop, comes alive and displays a sign: “Your fortune is being printed.” At this moment the receipt printer is initiated through the XBee and the receipt is printed. The small screen displays the word “Goodbye,” and the box returns to an idle state).


Because of the nature of the project, our main challenge was to make the installation both interactive and feature rich, as well as reliable. In order to deal with multiple functions we had to create multiple timers running in parallel, and use boolean variables as logical triggers. We had to create a number of small loops, which contained the states of different things within the main loop. Although the sensors and functions separately were quite simple, running them all together at the same time proved to be complex and we had to look for help organizing the code and figuring out the timers. Dimitri Levanoff helped us with working on the code, and we also asked our classmates and our TA, Ryan, for help when we unable to figure out the logic. We drew a lot of flow charts, which was very helpful.



After the code was written and most of our parts arrived Nick helped us to organize the wiring, which was very helpful. We used his diagram as a guide and checked with it till the very last moment.


Some parts of our code were still very unreliable at the time of the presentation and we would like to continue to work on the project until everything is figured out. The lights and the printer were the most problematic. We could not make the printer work reliably with the XBee and at the very last moment we had to install a small button, so we could initiate the printer by hand for the presentation. Initially we wanted to have two laptops – a main laptop in the booth, running the videos on the tablet, plus one inside with the projector running the processing. However, we ended up reversing it and using only one laptop inside.


The booth contained an Arduino, sensors, and the XBee. All of the video and sound were run on the laptop inside the booth and sent through the Xbee, which is a better option if we want in the future to install the booth on the street by itself and control it from a distance. We also ended up using Max instead of processing to run video and sound.


Pressure Sensor; proximity sensor and printer sketch:

zoltar sketch pressure-proximity printer_bb

XBee sketch:

Zoltar xbee sketch_bb

EL panel sketch:

board and shield



View the Arduino code on GitHub


Fabrication techniques:

Zoltar interface:

The initial concept of an interactive screen between the threshold of the outdoors and indoors began with a preliminary schematic of XBees sending data from a user interface triggered by motion, and a receiving XBee outputting an image projected onto a large surface.


Ideation sketches considered a feedback narrative of a linear sequence, starting from the user, leading to the projected image. The step thereafter is dependent upon the user’s reaction to the captured sequencing of the user’s actions.

One factor to note in this preliminary iteration was the deficit of an object/interface, which we as a group identified as an important aspect to the experience. The object/interface anchors the concept from start to finish, and  is central to the topographic systems map, thus it grounds the user interaction with a tangible node of access in connection to the visceral projected output of the story. The communal ouija board, seance table, crystal ball…part of the mythos of the medium’s power is the ability to communicate through a portal from this world to the next.

Initial ideas drew upon the attraction of mystery (the oft-cited “black box”), and the spectacle of amusement attractions. Fun houses, lights, sounds, wardrobes, and the prize/token reward at the end of the journey comprise the complete experience before (anticipation), during (interaction), and after (the memento/echo take-away).


Barnum and Bailey’s commodification of the experience:


Where even the exit becomes an event…

Thus, the temporal aspect and timing of the sequence became central to the mapping of ideas.


In this network topographic map, the user has a point of interaction to activate the event. As each stage unfolds, the mechanics of the interface scale up exponentially, adding layers of visual information through reactive LEDs, text, auditory cues, animated video and finally a Deus ex machina projection finale.


The orchestration returns to the tangible element scaled down to a receipt-sized paper fortune at the point of the user interaction.



The box required many levels of engagement in conjunction with ergonomic elements to consider. It was decided – as all tellers of fortune require – to have a palm-reading interface through which the user could activate the story. But first, the “black box” always has an entryway, visible from afar. The following sketch identifies how the user is captivated, by a ‘header’ signage piece (first conceived of as printed graphic, later etched), embedded LED lights, low-level audio cues, and a standing-height kiosk with full-colour graphics.


As the concept developed to include a tablet interface, a printer and a projection environment, the ergonomics and proxemics of the kiosk needed to be approximated to an environment yet to be determined. Though not entirely universally accessible due to height and block structure, measurements were based on averages of standing height of the class, accommodating a range of stature.

Multiple venues were explored, and in the end, an outdoor location on the corner of Duncan and Richmond street was chosen for the affordances of projection surfaces (windows) and outdoor proximity to the placement of the kiosk with oncoming pedestrian traffic.


Existing corner of Richmond and Duncan Potential viewing and projection surfaces.

The base of the kiosk was an MDF box (pre-fabricated) finished to withstand inclement weather, with an interior structure measured to accommodate the largest of the internal hardware: an HD car battery, required to power the components. The top surface was determined to be at an angle of 20 degrees to be readily visible, and to be a perceived affordance of a touch-activated interface.


Prototype kiosk sketches – base determined at 16”x16”x37.5”, with allowance of 5-6 inches of display top-mounted to box. Foreground kiosk renders the palm interface as an angled protrusion with a box-constructed backing for graphics, housing internal lighting and surface-mounted tablet/screen display. Background option simplified the design and eliminated extra material use, creating an integral unified housing with one affixed back panel, uplit with LEDs. The latter was chosen as the prototype to fabricate once the tablet had been established as the interface for the intimate stage of relaying instructions to the user (a computer screen was a second option). Red line rectangle represents a figure standing at 6’-0” height; centre of tablet to be affixed at 4’-8” – 5’-0”.


Once the surface-mount interface was established, the rules of engagement for the user to follow were discussed, and how this would play out. Initially, a set of buttons were going to be adjacent to the palm reader, which would help to categorize the type of responses the user would receive (and to aide in the suspension of relevant  belief). However, two sets of interactions visually and cognitively created complexities, and could have sent confusing messages at different times. It was then established that a graphic on a hand icon would support the action of the hand placement, which in turn would set the rules of engagement running on the tablet (“Now that we have your attention…”)


Final working drawing details of the kiosk before disassembling the pieces into a 2D format to CNC laser cut the individual parts.


Prototype graphic (colour option 2) + Zoltar graphic, inspired by the movie Big.
Palm graphic applied to the CNC cut die for the palm interface.


First two templates – surface-mounted palm reader interface – white opaque acrylic ¼” thickness, includes printer slot, IR sensor slot, audio perforations, cable management and access portal, friction-fit sides and allowances for heat-bent curvature with pre-drilled holes for mounting plates.


Third template – clear acrylic ¼” backing with etched graphic illuminated by edge-lighting white LEDs.
Surface-mounted detachable tablet template secured with hardware. Hand template cut within 10” radius, post-mounted with graphic, pressure sensor, and EL luminescent programmed light film.
prototype5Light testing with edge-lit LEDs on etched vector-based graphics, illustrator format.


Costuming & Props:

The costume and props for our fortune teller was curated from the collection of Tatiana Jennings, who runs a theatre company on Dupont Street in Toronto.



The casting for this project brought up some unexpected topics, including political correctness and cultural sensitivity. At the time of filming, the project was known as “Ask Swami,” and was to feature an moaning, overweight yogi. However, the actor that ended up playing our fortune teller was Caucasian, which led to some discomfort within the group. Some were concerned that a Caucasian actor playing someone who spoke Hindi may be construed by some members of the class as being insensitive.

At this point, the footage had already been filmed, and re-shooting the video would take considerable time. It was then decided that instead of “Ask Swami,” the project could be changed to “Ask Zoltar.”

Aspect Ratio:

Videos with different aspect ratios needed to be created, including:

  • Several widescreen (16:9) videos to display on the Samsung tablet affixed to the fortune telling machine.
  • A pillar-boxed (4:3) video to create a digital projection in the square window of the Graduate Gallery.

The aspect ratio didn’t just affect the final video output, but also the way in which the shot was composed while filming. The widescreen videos needed imagery that filled out the 16:9 space, while the pillar-boxed videos needed to be framed in a way in which his movements were confined to the centre of the screen.


The actor who had volunteered to play “Swami” was not fluent in Hindi, so it was decided on set that he could speak English and we would later dub in Hindi and include English subtitles. The actor then went on to dramatically recite lines from Shakespeare’s Richard III.

Later, as the project transitioned into “Ask Zoltar,” it was decided that our fortune teller should speak Romani (which is similar to Hindi, and is the traditional language of the Roma people of Europe, also known as Gypsies.) A YouTube search was then underway in order to find a clip of a man speaking Romani in an angry manner. Once the clip was found it was downloaded, and the speed was reduced to 65% in order to give it a deep tone with a menacing quality.

Sound effects were also needed, including the sound of an explosion, snoring and moaning. A nuclear explosion was chosen to go at the end of the last video on the tablet in order to announce Zoltar’s arrival on the overhead projection. Both the snoring and the moaning effects were pulled off of YouTube, with the moaning consisting of a “cat in heat” recording that was slowed down to 25% in order to match the tone with Zoltar’s voice.


The overall goal was to have the video on the Galaxy tablet and the large digital projection visually tied together through style and colour. We also wanted to give the footage a nostalgic feeling, similar to amusement games from the 1950’s.


All of the footage was altered with a green filter to give it a nice contrast against the clear acrylic on which the tablet would be mounted, and to ensure it stood out as passersby viewed the large digital projection from the street. Both the tablet videos and the projection video were layered with an “old projector” filter to give them both an old, eerie quality.



The pacing of the videos was determined by a storyboard that was drawn out to create a theatrical effect. Timing was crucial, as the successive videos were being triggered by sensors and communication between XBees.


Final Video Outputs:

Video #1: Black Screen with Snoring

Video #2: Eyes Closed with Snoring

Video #3: Zoltar Awakes

Video #4: Digital Projection

Video & Sound:

In order for there to be feedback between the user’s actions and the machine, we needed there to be a way for Arduino to communicate with some sort of video playback software. Processing would have been a good choice, but as the deadline approached quickly, Max came to the rescue, offering a visual way of connecting different inputs (the sensors), to different outputs (the video and receipt printer).



max patch

Full Max Patch.

Information from the sensors was written to the serial port using Arduino. However, the way it got to the serial port of the laptop indoors was by using two XBee modules. Max was instructed to look out for various messages in the serial port such as “1” and “2”. The first key message was triggered by the proximity sensor, and that told Max to play the first video of just Zoltar’s eyes. The pressure sensor under the hand triggered the next video which told the user to think of a question. And from there, the “clocker” object started a countdown to trigger the final video in the projector.
Once the final video completed, it sent out a message which could be further used to signal the receipt printer to print out the user’s fortune. But there were problems getting the message to be picked up in our serial port, which was very busy. As a result, a button was hooked up to help prototype this feature in.

The serial object brings in data from the XBee on the kiosk:
serialThe Route object sends various messages in the serial bus (0, 1, 2), to trigger specific videos. The Clocker object starts a countdown for playing the final projection.
The three videos are played using one Jit.QT.Movie object, and reading different quicktime files:
This is the projector video, which goes to its own Jit.Window for the second screen. A “loopnotify” message sends out a bang when the video is done to signal the printer to start.

Playback devices:

All video and its accompanying sound was sent from the Max patch. The goal was to keep everything in a central place to avoid sync issues. The second screen from Max was sent to the indoor projector using a VGA cable, and the tablet video was sent by sharing the main screen over wifi. In order to minimize wifi problems, an open router was placed right by the window. Sound was sent to a portable speaker over Bluetooth, and worked surprisingly well through the window.


The wiring underneath the palm interface.


Max 6 seems to have trouble with some HD videos. As a result, the video clips were highly pixelated.

The app running on the Android tablet, iDisplay, worked marginally fine indoors but through the window the choppiness was unusable. Even with a day’s notice that this program was going to have problems handling video streams, it became difficult to find a replacement remote desktop application quickly enough.

Finally, most traditional operating systems function on the concept of two screens. For this project, we would have required three screens: One for the Max patch, one for the projector, and one for the tablet. This only became apparent just before launching the installation, and so the main screen of the tablet was shared, which is far from ideal.


20141207_204835A still from the video is used to create brightly coloured graphics for the sides of the kiosk. 
projector1Testing the projector inside the Grad Gallery. Translucent white fabric was hung in the window to give the projection a surface to reflect off of. 
20141208_180024 Making last minute adjustments before the palm graphic is applied.
tablet_kioskAffixing the tablet and closing up the back of the kiosk are the last elements of the project to be finished.
guardingElliott guards the kiosk while we wait for the class to appear. Several passersby asked to have their fortune read as we waited.

Final Presentation:

night Zoltar appears in the night, within eyeshot of the CN Tower.
DSC_3251The tablet announces Zoltar’s imminent arrival.
projectionsThe XBee cues the projection from inside the Grad Gallery building. 
DSC_3289The group poses with Ask Zoltar after the class presentation.


Apple. (2014, November 8). IOS: Understanding iBeacon. Retrieved from

black automaton (n.d.). Fortune Teller Lady Machine. Retrieved December 14, 2014, from website Tumblr:

Crystall, B. (2007, July 5). A programmable robot from 60 AD. New Scientist. Retrieved from

Dasilva, D. (2014, October 10) It May Be the ‘Now’ Economy, But Is Main Street Retail Ready for iBeacon? Retrieved from

Deutsche, R. (1996). Evictions: Art and Spatial Politics. Cambridge, Mass.: MIT Press

Gilmore, J. & Pine, J. (2007).  Authenticity: What Consumers Really Want, Boston: Harvard School of Business Press.

Groen, D. (2012, February 9). Who controls the CN Tower’s coloured lights? Retrieved from

Hess, R. (2014, January 7). Building a Working “Zoltar Speaks” Fortune Teller. Retrieved from

Kleiman, J. (2013, August 30). Cedar Fair Parks Take Guests Into the Mountain and Up to the Skies in 2014. Retrieved from

Koetsier, T. (2001). On the prehistory of programmable machines: musical automata, looms, calculators. Mechanism and Machine Theory, 36(5), 589–603. doi:10.1016/S0094-114X(01)00005-2

Lau, C. Research: Interactive installations. Retrieved from

MacDonald, B. (2014, February 24). Innovative roller coaster-dark ride combo heads to Canada’s Wonderland. Retrieved from

Martin-Robbins, A. (2014, May 23). Vaughan’s Canada’s Wonderland set to unveil interactive ride. Retrieved from

Palmeri, C. (2014, March 7) Disney Bets $1 Billion on Technology to Track Theme-Park Visitors. Retrieved from

Raphaël von Allmen Design & Scenography. (2009). Helvetia Park. Retrieved from

Safran, L. (1998). Heaven on Earth: Art and the Church in Byzantium. Pittsburgh: Penn State Press.

Sick of the Radio. (2011, April 13). ART:Helvetia Park, an installation in Switzerland. Retrieved from

Scotiabank Nuit Blanche Toronto. (2012). Funhouse, 2012. Retrieved from

Triotech Studios. (2014). Canada’s Wonderland. Retrieved from