## Brief Overview of Artificial Neural Network Systems

Last week I gave myself a crash course on Neural Networks and Machine Learning in an attempt to disseminate the concepts behind the core algorithms and processes behind artificial neural networks, machine learning and agents of artificial intelligence. The topics are often discussed from a mathematical perspective; I wanted to review a set of readings and make my own interpretation of what I found in terms of programming terminology for the sake of applying the concepts in my research.

Artificial Neural Networks (ANN) are assemblies of artificial neurons, sometimes called units, which are not designed to imitate actual neurons but modeled after biological findings of how the brain works. We are not quite at a stage of scientific discovery where we can say with certainty how neurons work individually or within a network in the brain but scientists have nonetheless created digital models of their theories of how neurons communicate. These ideas have carried over to computer science as inspiration for how to develop and model algorithms that seemingly have the capacity to perceive, learn, and make decisions based off of sensory data.

`Above: a model of the neuron (Cichocki)`

There are many artificial models of the neuron, but the general idea is that neurons take in a numerical array of information (often binary) and pass the array through a weighted threshold  in order to determine an output value. The output is also represented numerically, often as a value between 0 and 1 or -1 and 1, either in binary and/or analog format. The output is dependent on the weighted value threshold, which determines how ‘confident’ the neuron is that the input passes or fails its threshold (e.g. if at least 60% of the inputs are 1’s, output a 1 signal). In some models, the output can be fed back to a weighting algorithm within the neuron to determine whether the weighting threshold should be modified and to what degree.

```
Above: ANN arrangements: a) FeedForward b) FeedBack c) cellular arrangement (Cichocki)```

Neurons can be arranged in a number of ways within an Artificial Neural Network. Their outputs are fed to the inputs of other neurons which can arrangements such as the examples above.  FeedForward is essentially a chain of neurons, while FeedBack incorporates neurons dedicated to returning incoming information back into the chain. (Artificial) cellular arrangements involve neurons with multiple, nonlinear connections with each other.

In many neural networks, these chains occur in layers such as described in the feedforward multi-layer Perceptron model above, one of many models used for designing neural networks. Multiple neurons with very similar input domains and tasks are arranged to communicate with neighboring layers but not neurons on the same layer. These networks can have multiple layers, but apparently run most effectively with three layers for most purposes. The layers themselves do not have to have the same number of neurons as any of the other layers. This arrangement of neurons allows the Perceptron to analyze data chunks instead of individually at a ‘pixel’ scale.

For example, in an image recognition process a specialized neural network might be tasked to analyze the direct capture of data from a camera in one layer, (possibly split across red, green, and blue channels), determine small clusters where brightness values form discrete lines or patterns in a second layer, then determine if these clusters form a particular shape in a third, then finally feed this information to a single neuron to determine how closeley the collective analysis from the third layer matches a ‘learned’ model for a cat (many machine learning algorithms undergo a ‘training process’ to prepare a neural network model to recognize certain features; those that do not are considered ‘unsupervised machine learning algorithms’).

In essence, ANN’s use the collective processing power of many smaller units in order to solve higher level problems. The fact that these machine learning algorithms are being brought up in our Body-Centric class is rather eye-opening with regards to how such technologies might be integrated with bodily technologies. A friend of mine referred me to an example of neural networks being integrated with prosthetic, where the prosthetic limb interpreted incoming electrical signals and gradually learned to perform actions reflective of its user’s intentions. If I remember correctly, I believe he was describing imitative machine learning, where the machine tries to imitate particular ‘memories’ that are implanted in it during its training process; perhaps such a device could imitate human hand motions. I wonder if and how some of these concepts could be carried forward to some of the sensory technologies we experimented with in the past, including vision, the EMG sensor, etc.

Sources:

Cichocki, Andrezej. “Neural Networks for Optimization and Signal Processing” Chchester, NY: J. Wiley, 1993. Print.

Buduma, Nikhil. “Fundamentals of Deep Learning” Sebastopol, CA: O’Reilly Media, Inc., 2017. Print.

Castano, Arnaldo Perez. “Practical Artificial Intelligence”. Apress, 2018. Print.

## Haptic Workshop Notes

Tyson Moll

This week’s task: try out three different experiments based on our class outline for haptics and record our findings.

Experiment 1: Hello Vibe Motors

In this experiment we were tasked with testing out the basic functionality of LRA (linear resonance actuator) vibrating motors  with Arduino sample code.

With the basic digital “Blink” code, we saw the motors performing at high intensity. Switching to an Analog output allowed us to change the intensity of the vibration and adjust the intensity over time. Decreasing the delay between vibrations seemed to make the motors feel less intense but made the on/off pattern more distinct. The “Fading” example code allowed us to play with the intensity of the vibrations with analog output.

The vibration felt vaguely uncomfortable on more sensitive regions of skin, such as the back of my hands and around the lips. The intensity felt similar on my fingerpads, albeit not uncomfortably so. Testing out the vibrations on less sensitive areas such as cloth-covered legs and regions of my arms made the vibrations feel less noticeable and bothersome.

The difference threshold for me was an analog output value of about 40 (out of 255); I had to close my eyes and concentrate on the motor to detect its action, which reminded me of some meditation teachings that my cognitive science professor once showed us… focusing on an element of the body  Any lower and it became difficult for me to determine whether I felt anything besides my own pulse.

Experiment 2: Motor Arrays

For this experiment we used several motors in order to note any differences from single-motor sensations and see if we could detect any haptic illusions.

With two and three motors vibrating in .2 second cycles, it created a pulsating effect that felt as if a single force was pushing against my thumb in a discrete direction.

Using the “fading example” made the rising/fading effect overall more noticeable. I tried having two motors fade with opposite values (e.g. 12% and 88%) and it made the vibration seem to move. Interestingly, it seemed as if this motion was more noticeable when the motors were positioned horizontally relative to my body than vertically; positioning the motors on my fingers across or along its length also had a different effect (especially when the motors were placed on separate pads of my fingers), but it didn’t seem as discrete as changing their relative position to my chest. Maybe I am misinterpreting this feeling but it could have some sort of connection with the body’s ability to sense the relative position of body parts.

Increasing the speed of this pattern created a pinching effect whenever they changed from increasing to decreasing vibrations. My guess is that this is caused by the hands detecting a sudden shift or delay in the pattern of vibration. The apparent back and forth movement was also more apparent.

I also created another version of the ‘fading’ code with three different offset values for each vibration motor, but the results were vague and difficult to interpret when in contact with several body parts.

Without a physical bonding apparatus we found testing out the motors easiest when we squeezed them between our fingers securely. I later tried out masking tape as a securing apparatus. Attaching the motors to my desk surface made an effect that resonated not unlike vibrating cellphones on classroom desks: the effect carried across the surface, although the patterned action was less noticeable. I also tried attaching two of the motors to the back of my shoulder and triggered two successive, slightly fading ‘pokes’ using relatively strong pulses.

Experiment 3: Haptic Motor Drivers

For the final experiment we had the opportunity to try out Adafruit’s haptic motor drivers (HMD). In preparation, we downloaded the relevant library through Arduino’s library manager and opened up a basic example code that cycled through all of the drive modes on the HMD’s chip. Some were gentle, some sharp, some subtle. I’m not certain I can say I have a favorite effect or that any of them are of particular interest but being able to readily access this variety of options via a chip seems convenient. I did find the transitions between the hums and other similar behaviours very streamlined, however; the idea of being able to smoothly chain these behaviors in sequence seems like its most useful trait.

The last task we were assigned was to represent how our day went yesterday as a series of haptic buzzes, paired with some sort of sensor. Instead of developing such an arbitrary representation, I wired up the vibration driver with the ultrasonic sensor I covered in the previous experiment and added a few additional thresholds, resulting in a device that increases or decreases its rate of pulsing based on how close an object is to the sensor. It reminded me of the use of rumble in Nintendo Gamecube controllers, especially in ‘hot / cold’ scenarios where approaching items of interest would increase the rate of vibration. I suppose this could be handy as a means of navigating while blind; it could even be translated into some sort of maze-navigating game.

In conclusion, haptics provide an underused means of sensory communication in a world dominated by visual and audio cues and haptic illusions are quite exciting in their ability to convey information through simple repeated movement.

## Peripheral Sweat Bands

Tyson Moll

[Arduino Code: PASTEBIN.com]

Today we worked with themochromic pigments in water and acryllic paint applications. The pigment changes from a coloured tone to white under heat and electric current as typically seen in novelty colour-changing mugs. My favorite example of the technology remains to be the CD version of the Nine Inch Nails album, Year Zero: when the CD is retrieved from a music player, the normally black disk becomes white and reveals hidden detailing.

Year Zero, Before and After [Source]

For the purpose of this workshop my idea is to use thermochromic-dyed sweatbands to indicate whether or not there is activity in a sports player’s blind spot within a particular distance. I imagine this could be accomplished with ultrasonic sensors, tuned to activate the bands whenever activity is detected within 10 feet of the user. Using two of such devices could bring an extra level of sensory detection that is visible within a player’s peripheral vision, providing warning of an unseen action before it happens, whether it be an opponent or an obstacle.

For the purposes of conserving material, I decided to prototype this with the existing fabrics we created in class rather than purchase sweat bands for this one-use project. The programming was prepared with an Arduino and code that I slightly altered from an example for detecting proximity with the ultrasonic sensor (included at the top of this post). I sewed the thermochromic material with  conductive thread and connected it to my circuit circuit: the device worked as expected, although based on testing the effect was more immediate with higher voltages (ergo, more heat caused by the electric current). When the sensor was tested with a detection threshold of 10 feet, it behaved somewhat finnicky but was certainly activated enough to prove that it could work; signals that came into contact with fabric, reflectives and non-perpendicular angles caused irregularities. I’ve heard there are more reputable sensors available, perhaps worth looking into beyond the prototype stage.

Naturally, were the device to advance to a proper prototypical state the sweat bands would need a means of communicating readily with the sensor without it being mounted directly to the sweatband (wireless?) and the thermochromic pigment would have to be modified to properly imbue itself in the sweat band with resistance to moisture. The voltage would have to be high enough to  have the effect be noticeable within a certain timeframe, though we have also been warned about running the device for long hours due to the amount of heat generated by the circuitry.

I don’t think that the material as-is is ready for such an environment, but I do think that thermochromic plastics would be effective paired with proper electrical regulation and protection mechanisms. All and all, this was a fun little excercise and I’m glad to have had the opportunity to try out the materials.

## Finger Activity Tracking with the PulseSensor

Tyson Moll

Code: “GettingStartedProject” with delay set to 100ms
Hardware: PulseSensor circuit to record data as stipulated for above example. See https://pulsesensor.com/pages/code-and-guide
Software: PuTTY (serial logging to .csv), Excel, Overwatch

This week, my task is to find a good use for a PulseSensor; a device that uses a light sensor and green LED to determine your heart beat.

At first glance, I wasn’t totally sure how I could use the sensor beyond translating a beat into sensory phenomena. Strictly using the device as a heartbeat monitor did not appeal to me, so I began to explore ways that the device could be used differently to track bodily behaviors.

Turns out it also works relatively well as a speed radar! Since the light sensor is measuring changes in light, the device effectively can be used to determine how fast an object passing over the sensor is with ‘BPM’. It might not be the most standard unit of measurement, but it inspired me to observe ways I could accurately measure using the device.

In a previous project I worked with Ultrasonic sensors to measure the distance between it and objects. One of the notable quirks of the ultrasonic device was that it required a relatively flat surface to reflect its signals off in order to function most accurately. The PulseSensor tends to behave in a similar fashion, working best when the sensor is aimed upward and away from objects.

I tested out the sensor inside of the finger-sized device I made for my previous blog post and it was relatively successful in tracking the activity of my finger as I moved it, so I thought it would be interesting to better secure the device inside the “finger pocket” and record the data it collected into a spreadsheet. I wasn’t entirely sure whether the device would be accurately recording the behaviour of my heart or the movements of my finger but I was interested in studying the analog behavior nonetheless.

PuTTY presented itself as a viable solution, having the option to send serial data to a printable output log file; all I had to do was send it to a .csv instead of a log file and open the .csv with Microsoft Excel in order to retrieve the data output. For the sake of getting an accurate reading without too many data points, I chose to use the “GettingStartedProject” example in the PulseSensor arduino library with a serial delay of 100ms, which provides the value of the sensor as received by the Arduino’s analog input pin.

I recorded the data with PuTTY in two sessions: one consisting of relaxed typing activity comprised of cataloging records in an online music archive and another consisting of activity recorded while playing the computer game Overwatch in its Deathmatch game mode.

I plotted the information I collected into the line graphs above and below, clipping data collected when setting up the device and restricting the view to information between analog values 650 and 350 (the Y-Axis). The numbers along the X-Axis relate to the sample number and constitute about 5 1/2 minutes of time (since each sample is recorded every 100ms). The graphs seem to indicate that Regular Activity (RA) values gravitated towards an analog value of 520, whereas Gaming Activity (GA) values tended to be slightly lower on average. However, GA did exhibit more consistent spikes in activity as well as much more obvious fluctuations, waving up and down versus RA’s relatively flat behaviour. While RA did have several periods of high level spikes, it seems likely that they do not act as clear representatives and could possibly be due to some human error on my part (sometimes the device was adjusted during this first trial, and the end of the RA data constitutes taking the device off).

In order to get a better view of this data behaviour, I overlayed the two graphs and reduced the information collected to a representative sample between X values 2100 and 2600, which seemed to further support my understanding of the behavior of the recording data.

Ultimately, this experiment provides visual insight into data that a user may not be able to examine while preoccupied with an activity. Although this data is largely technical in scope, it does leave me imagining ways of using data collected via bio-metrics that does not express itself instantaneously and rather leaves its findings to be reflected upon at a later time. The obvious comparison would be to medical or sports analysis devices applied to a user, but I am interested in how such devices integrate in ways that do not interfere with their analysis and how they might move beyond the sterile environment of scientific analysis.

Perhaps such a device could monitor and save data over a period of time (with an SD card instead of a serial connection) and if it crosses a particular threshold over a sustained period of time create some sort of alert that may be informative to a user. Especially if this device is a more long-term use product such as an insulin pump, it could be interesting to explore how such devices could be better adapted to everyday human use.

## Stretch Sensor for Digits

Tyson Moll

The task for this week’s project was to develop a ‘body-centric’ technology that used resistive material.

Investigations:

During class, a classmate and I tested out five different resistive materials using both an analog multimeter and an Arduino programmed to receive analog voltage input with values within the range of 0-1023. In some cases we used alligator clips directly on the material, and in others we wedged the resistive material between two strips of conductive material attached to neoprene, to allow us to press into a soft material instead of, especially in the case of the plastic sheets, material without much ‘give’.

It was interesting to compare readings with different testing methods on the materials we were provided. Sometimes we stretched the materials out, sometimes we replaced the resistors used in the arduino testing with other resistive values. The resistive values of the materials tended to reduce when in tension or compression; the closer the fibers (and particles in the case of the plastics) were to each other, the less resistance they exhibited. Similarly, voltage readings with the arduino were higher when there was less travel distance between the two clips attached to the materials.

Table 1: Material Tests with Multimeter

Table 2: Material Tests with Arduino

(Above: investigative tests of the material properties of several resistive materials)

The Device:

For this iteration, I decided to to try crochet with the help of my girlfriend. In my last exercise I learned to use some crochet techniques in order to bind two edges of knitted material together. From what I’ve heard, the process of crochet is faster than knitting with practice which appealed to me. The catch was that the process was slightly more nuanced; to create the knots, the crochet hook  must be weaved in and out of the material in more steps than the basic knit. The hook of the tool makes this process relatively painless though, and in a matter of minutes it felt as natural as the knitting process.

What resulted was a crochet cylinder roughly large enough for my thumb. I attached the stretchy resistive fabric with non-conductive thread in a manner that left the material in tension, then sewed conductive thread to the short edges of the strip, leaving two exposed strands at the bottom for the purposes of circuitry. Similar to the previous blog post, I wanted to explore handheld activation mechanisms, this time with the sensitivity of an analog input to Arduino.

Testing:

I hooked up the device to an arduino programmed to take analog read input and feed it through the USB serial connection to the computer for display. I tried several on hand resistors to see which one would be most effective for the task; none of them were ‘amazing’, but the 4.7 ohm resistor paired with the arduino’s 5V power supply provided clear differences between the device being flexed and relaxed.

Next Steps: Further calibrate the resistance using the multimeter to get the largest sensitivity of data! And make more interesting devices.

## 2019: A Knitting Odyssey

2019: A Knitting Odyssey

Tyson Moll

The task for our body-centric class this week was to develop a component or element for a conceived design using two instances of either felting, weaving or knitting techniques. In my case, I chose to pursue felting and knitting.

Strategy:

My initial design concept was to create a fabric dress that one could retreat inside for privacy; a wearable tent of sorts. This concept simplified to a mechanically-retractable hood, then to gauntlets that could provide coverage depending on whether a hand was clenched or not. The devices I chose to design were to act as elements of the larger idea of realizing the gauntlet. As the glove would need to be activated in some sort of manner, I focused on developing switches activated through tension or compression. What resulted included a fingertip- activated pinch switch that could be integrated into an interior mitten to activate an external closure and a felt ‘stress ball’ activator that could be positioned within the palm to activate the gauntlet (and otherwise dangle like a pom-pom from the wrist).

With my unfamiliarity with fabric circuitry, I wanted to keep my components relatively simple and base my designs off of examples shown in class. It was hard for me to envision any of the techniques as critical components to the design of the device as my heart was set on working with a fabric that could be easily cut to shape based on unrolled geometry from Rhinoceros, but having now worked with the materials and techniques in question I feel more confident in the potential to use and incorporate more analog methods into my concepts moving forward.

KNITTING

For the knitting component, I initially decided to make a knit button based on a design shown in class consisting of two conductive layers and a middle non-conductive layer, activated in compression.

Specifications:

• Approximately 20 loops in length and 8 repeated knit loops, with one side folded over and turned into a fingertip-sized pocket using a crochet hook to secure the edges
• Consists of knitted yarn with sewn conductive thread integrated at the fingertip and ‘palm’ patch of the device.
• Tools include knitting needles, a conductivity testing device and a crochet hook.

Process:

The last time I knitted was at least a decade ago. Familiarizing myself with the process again took a solid 30 minutes of watching a 10-minute YouTube tutorial. Becoming more comfortable with the process felt like a bit of a double-edged sword as I would occasionally forget where I was in the process of the technique if distracted or mindlessly repeating the loops.

But I picked it up the next day and watched 2001: A Space Odyssey for the first time… which is possibly the best film to watch in the middle of knitting.

The knits went smoothly, but slowly. Thanks to my incredible character trait of laziness I imagined ways that I could create something interesting with the small piece I eventually would cast off. One of the first ideas that came to mind was to attach a second element to the midsection of the strip and create a pinch switch activated when the three extensions of this material would be in tension. This led me to look up how knitted pieces could be attached after being cast off, which turned out easiest with a crochet hook. This in turn inspired me to consider using the existing knit strip over top my fingertip; I wouldn’t have to create an extra piece and could use the tip as a component in a pinch switch.

Nice.

Maybe if I had the foresight I would’ve sewn the conductive thread into the knit material prior to knitting the conductive elements but a simple star-sewn patch did the job suitably for prototyping purposes.

FELTING

With the felting process I decided to create a simple pinch pom-pom based on the examples. Why not just activate the gloves with a ‘stress ball’? The idea to dangle it from the sleeve of a coat or the device came later.

Specifications:

• Consists of felted white wool (approx. 2” spherical diameter) and 10” of conductive wool
• Felt two ends of the white wool, leaving the centre unfelted. Poke a hole through the middle with your finger, then apply half of the conductive wool via felting in the two unfelted parts of the white wool. Felt the remaining wool, being sure not to connect the two patches of conductive wool together, until you create a sufficiently firm sphere.
• Tools include a conductivity testing tool, a felting needle, a firm sponge, and a fingertip protector

Process:

I stabbed a puffy ball of wool with a felting needle until it firmed up on both ends, then established conductive felting on both sides of the midsection of the woolly ball. I tested the device to see if it worked throughout the process; definitely most effective when fully compressed properly based on the manner I injected the conductive thread. Squish-n-go!

I would be interested in trying again but with the conductive thread intermingled throughout the entirety of the wool in the future but I wanted to go with a design that I knew would work for my first attempt at felting. The method seems very effective as a means of concealing the circuitry as the felt seems very accommodating to the potential of it being stuffed with logic chips or the like (I would have to test how effective of an insulator the wool is and whether I could make it accessible for repair).

End Notes

Working with these techniques took me out of The Comfort Zone and into a realm I have always been excited and interested in but never brave enough to summon the initiative and work with.

References:

In addition to lecture notes provided by our instructors, I reviewed the following YouTube tutorials to learn how to do the techniques used in these devices.