MindPainter

img_0722

Overview

MindPainter reads a user’s brainwaves and uses them as the foundation for a painting. Each painting is unique to the user and the moment the readings were collected.

collage-01

It can also be used as a tool for meditation, with the inclusion of a meditation mode that measures the user’s mental relaxation and adjusts the image accordingly.

The shape of the image is determined by a mathematical function called the Rhodonea Curve.

rcurve

The Rhodonea Curve as described by Wikipedia

The Rhodonea Curve can be applied to draw shapes often described as “roses”, as well as to the game Spirograph.

rose-curve-demo

MindPainter expresses the Rhodeona Curve as code in p5.js and Processing. By adjusting variables in the equation, the code allows users to create mind-bending fractals.

The colours of the elements in the drawing are determined by assigning the values from sensor readings to RGB parameters.

firstcode

The basic code in p5 and Processing for generating Rhodonea Curves.

test5

Potentiometer inputs for k and j in the code above created this image. 

The Mindwave hardware returns a value for the following readings:

Delta
Theta
Low Alpha
High Alpha
Low Beta
High Beta
Low Gamma
Mid Gamma

Though some claim that brainwaves have significance it is not possible to draw meaningful conclusions on the relations of these readings to thoughts or mental states with the MindWave, as the hardware is actually reading electronic activity on the surface of the skin and extrapolating that data to speculate on brainwave activity (see MindWave Mobile user guide).  However users of MindPainter have reported being able to see relationships between their thoughts and the paintings MindPainter generates – for example, in one test, a user reported having an angry thought which led to a picture that was red and jagged and spoke of anger to the user.

The Mindwave also captures two proprietary values that are not based on brainwave activity but are extrapolated by the hardware:

Attention
Meditation

In testing it was determined that these values tend to be more reactive to user intention than the brainwaves.

The default mode of the MindPainter software uses all the values in order to create a series of notably differing images.

The secondary mode, Meditation Mode (accessible by pressing the ‘m’ key at any time), uses the brainwave readings only for generating cosmetic elements such as colour. The actual drawing of the shape is handled only by the Attention and Meditation parameters, meaning the image can be clearly related to the mental actions of the wearer though it always draws a starburst shape. The larger the starburst, the deeper in a meditative state the user is.

Concept

img_20181111_220548

The initial concept for this project was a participatory meditation. One user would wear the headgear and sit in meditation while other their brainwaves were visualized on a screen. In this way their meditation could become a group activity and an act of creation as well as introspection, as others could view and participate in it.

Another iteration of the early concept was as a meditation assistance tool. The user would sit in meditation and if their attention wandered an Arduino would cause a solenoid strike a chime, bringing the user’s focus back.

During development the concept moved away from being focused on meditation and into the more accessible artwork-generation program that it became.

Process

The development of MindPainter was a triumph of research more than anything else. With the understanding that all sensors simply return numbers I wanted to explore interesting and unique sensors and the possibilities they could provide. I took some time to consider the sensors available to me and remembered that there were sensors that could take brainwave readings!

Before committing wholeheartedly to the idea of using brain sensors I experimented with the sensors available to me. A fast and simple way to express sensor data is visually, so I approached the problem as an art project. Since sensors return numbers, how might I express those numbers in an interesting visual way?

Some research into painting by algorithm uncovered the Rhodonea Curve, and a video from our friend at the Coding Train provided the basic code that would become the core engine of the painting tool. I rigged up an Arduino with an array of sensors and experimented with it to see what kind of images I could generate.

mindpaint-sketch_bb

The code for replicating the MindPainter imagery via sensor and Arduino can be found on GitHub

The Arduino proved the concept: sensor readings could be used as inputs to generate complicated, visually compelling, reactive images.

At this point, secure in the notion that I could reliably generate interesting images with constant sensor data, I went looking for an affordable brain sensor. The least expensive option available on short notice was the NeuroSky Mindwave.

img_20181031_144452

NeuroSky has been making brainwave sensors for some time and there is a fair bit of documentation available for research and hacks. Thus the hacking documentation I could find was for previous models of the NeuroSky hardware. Nevertheless, I wanted to experiment with the hardware.

Documentation I found early on suggested that the sensor readings could be accessed via Arduino by soldering wiring directly into the TX and Ground pins of the Mindwave’s chip. Despite having a newer model than the documentation called for I wanted to try it and see if it would work.

img_20181031_145328

I took apart the Mindwave, made the necessary solders, uploaded the Arduino code for reading the sensor data, and tried to get the connection between Arduino and Mindwave working.

img_20181101_222530

img_20181101_222523

While the code compiled and loaded to the Arduino, there was no transmission of data between the devices.  I did more digging. I confirmed that the hardware that the documentation I was working from was written for was not the same hardware I had, but an earlier model that transmitted its data via USB. The model I had was the Mindwave Mobile, which used Bluetooth. Having never worked with Bluetooth before, I had no idea how to progress. Clearly this was going to be more complicated than soldering a few wires.

I got a hold of a HC-05 Bluetooth Module and did a few experiments. I tried to set up the Arduino as a Bluetooth receiver to bypass the dongle the Mindwave connects with by default. I tried to set up the Arduino as a transmitter to replace the Bluetooth inside the headset. I even managed to find documentation that was, seemingly, exactly what I was looking for: connecting HC-05 to the Mindwave Mobile. However, due to my own unfamiliarity with the hardware, I could not get the Mindwave and the Arduino to interface. In the interest of completing the assignment in time, I was ready to shelve the hardware interface and produce something interesting using only Arduino-based sensors and the painting function.

As a last-ditch effort to get something meaningful out of the brain sensor I sat down with Nick Puckett who advised me that I was trying to take too many steps. He explained to me that since a computer is orders of magnitude more powerful than an Arduino I’d have a far easier time if I looked for a method to connect the headset to the computer directly and skip the Arduino altogether. Furthermore, he said, it was very likely that someone had already written the code for connecting the Mindwave to a computer, and that I just had to find and adjust it to suit my purposes.

The hunt was back on. I went back to digging. I wasn’t able to find anything that attached the Mindwave to p5, the language we had been working in up until now, but I did find several options that used Processing (which is similar to p5). Not being constrained to using p5, I decided to eschew it in favour of Processing, since that was where most of the work of interfacing the headset with the computer had been done.

Finally, after several attempts to implement and understand the code I had found, I came across a post on the Processing forums. A user had a problem similar to mine – they needed code to parse the information coming through the port the MindWave was using – and ultimately, they solved the problem. They didn’t use a large custom library to read the data, but only a few lines of code.

This was the breakthrough I needed. This was a simple solution that allowed me to quickly comprehend the work necessary to translate the sensor readings to useful data. The code, when implemented in Processing, worked. For the first time I was able to connect the Mindwave to software besides the apps it arrived with. I ported over the code from p5, rewrote it as required for Processing, and connected the sensor. Below are the first images I generated using my brainwave data:

processingtest

processingtest2processingtest3

From here it was a matter of experimentation. I quickly realized that the more complicated the object drawn the slower the program would run. This coupled with the slow update rate of the hardware led to the decision to make the final MindPainter a sort of slideshow, showing a new image every two seconds of the data that was last available. I experimented with the placement of sensor data in various places in the draw equation until I settled on an arrangement that seemed to generate the most reliably interesting shapes without annihilating the frame rate.

This experimentation gave way to some quality-of-life features that I implemented both to make my experimentation process easier and to create features an end-user might want. First I implemented a system that displayed a note at the top of the screen when there was a connection issue, as I often was unsure if the shape I was seeing was due to the hardware having a bad connection or me having low brain activity. Next I implemented a feature that displayed the current brain readings with a keystroke (‘n’ in the current iteration of the software) as I (and future users, I expected) wanted to be able to see what kind of readings were generating the pictures I was seeing.

Finally, since I still wanted to explore the software as a meditation tool from the earliest concepts, I implemented the meditation mode. One arrangement of the sensor readings had not generated the kind of psychedelic shapes I was looking for, but it had generated a consistent highly reactive starburst shape based on the Attention and Meditation values. Staring at the centre of the starburst had a calming effect on me, and it seemed that as the readings rose so did the size of the shape. I wrote a function to swap between this arrangement and the default one.

prettyones2mindpainter135158mindpainter58351mindpainter29850

Images painted by the final software

Next Steps

If I expand this project (and I plan to) I would like to implement a user interface that allows wearers to swap the parameters they are tracking on the fly. With ten sensor readings and dozens of possible places to put them in the equation, I would love to give a user the freedom to experiment with where the readings go and what they do. In this way a user would gain a far finer degree of control over the kind of images they generate than they currently have, and perhaps allow for an increased sense of ownership and artistic pride in the images they create.

I envision this project as an installation where the images generated are projected on a sphere or the walls all around the subject, and where sensor readings also cause chimes to be struck or music to be played. I envision a complete audio-visual sensory experience generated by the user’s brain.

I would also like to optimize the software so it runs smoothly, and even go so far as to implement animation so the image is constantly in motion.

Context

As the ability to measure brainwaves has become accessible it has become folded in with the trend of tracking biometrics. Companies such as NeuroSky and Muse are offering products to measure and track brain activity (as well as some ancillary apps such as games). These companies are relatively innocuous and are promising nothing more than the ability to track your own numbers, like a FitBit for the brain. Some products, like the NeuroProgrammer 2 referenced in this article, are utilizing similar technology and flowery language to back up scurrilous claims of medical benefits – possibly to the detriment of their customers.

In making MindPainter I hope to fall in the former camp. I make no claims about the benefits of the program. In generating works of art rather than graphs or arrays of spreadsheets, I hope that a user of MindPainter might develop a relationship and sense of ownership and pride in their brain readings. If the goal of Muse, NeuroSky, and mindfulness apps such as Calm and Headspace is to get people thinking about and familiar with the activity of their minds, then MindPainter is a tool in the same tradition.

Special Thanks

Built with NeuroSky MindWave Mobile. This project was inspired by the work of Arturo Vidich, Sofy Yuditskaya and Eric Mika from Frontier Nerds and the Mental Block project http://www.frontiernerds.com/brain-hack

This project would not exist if not for some excellent advice from Nick Puckett.

While I didn’t use his code, some insights from the work of Eric Blue pointed me in the right direction.

Code from user precociousmouse from the Processing forums provided the breakthrough I needed. Thank you, whoever you are.

References:

Brainwave Entrainment and Marketing Pseudoscience. (2008, May 29). Retrieved from https://theness.com/neurologicablog/index.php/brainwave-entrainment-and-marketing-pseudoscience/

The Coding Train (2017, February 08). Retrieved November 12, 2018, from https://www.youtube.com/watch?v=f5QBExMNB1I

Experience Calm. (n.d.). Retrieved from https://www.calm.com/

Frontier Nerds: An ITP Blog. (n.d.). Retrieved from http://www.frontiernerds.com/brain-hack

Meditation Made Easy. (n.d.). Muse. Retrieved from https://choosemuse.com/

Meditation and Mindfulness Made Simple. (n.d.). Retrieved from https://www.headspace.com/

Mindwave Mobile 2 (n.d.). Retrieved from https://store.neurosky.com/pages/mindwave

Mindwave Mobile: User Guide http://download.neurosky.com/support_page_files/MindWaveMobile/docs/mindwave_mobile_user_guide.pdf [PDF]. (2015, August 5). NeuroSky Inc

What are Brainwaves? (n.d.). Retrieved from https://brainworksneurotherapy.com/what-are-brainwaves

Project GitHub: https://github.com/npyalex/mindPaint

More photos: Album on Google Drive