ML5 – Self Identification with MobileNet


The term ML5 and TensorFlow and PoseNet has been loosely tossed around in the DF studio nearing the end of this semester from my peers in Body-Centric Tech and perhaps I should be feeling lucky to finally dive into it, regardless of the stress from living deadline to deadline ūüôā But, snipes aside, this was a whole other realm of knowledge that was¬†extremely new to¬†me, and I decided that by getting a better sense of what ML5, etc. is, that I would find its true appeal. Below are¬†key bullet points I made for myself:

  • ML5 – Machine Learning 5, in which it is symbiotic to P5.
  • ML5 doesn’t really need p5… but it makes things easier for you, in order for you to use Tensorflow, the¬†mother to Tensorflow.js.
  • ML5 deals with a lot of pre-trained models.
  • ML5 library doesn’t know a lot about Humans.

After looking at all the juicy examples, I wanted to explore the capabilities of the Image Classification with MobileNet, since it is “trained to recognize the content of certain images.” I wondered to what extent is MobileNet been trained to, and how? In what context? How have its creators trained it, and Who were they? What kind of MobileNet am I tampering with here? Has it perhaps been… westernized? Or just based on completely obscure datasets?

With¬†my burning questions, I played around with the ml5 index page’s example with the drag&drop/upload an image. Below are some of my discoveries when it came to images¬†available on my desktop.


Male Japanese Celebrity is a diaper/nappy,¬†A¬†Sim from the Sims 4 is a bathing cap apparently, and the Poliwag from Pok√©mon is a CD Player. Great! A lot of the ML confidence decreased, when it came to anything with human-like features. I¬†continued to upload a lot of other screenshots I’ve taken from animated/drawn media and it came to an interesting result:


Game Fanart, Spongebob and A cover from the Case Closed manga were ALL considered Comic Book with varying degrees of confidence. I started to realize how even images that are completely different can all be classified under one type of image. From there, I decided to upload pictures of myself (Yes, I volunteer! As Tribute!) because I was entirely curious as to how MobileNet would classify by appearance.


Surprisingly, I am Abaya. The funny thing about MobileNet classifying this as Abaya is how completely specific this word is to the middle-eastern context, where culturally, women of the Arabian Gulf would wear a black dress/cloak called an Abaya in their daily lives. I would think that only the first image could be classified Abaya due to the black material I am wearing on my head, but then it did it for my lighter colored one. Regardless of its confidence in recognizing me, there are varying percentages of how sure it defines the head scarf as Abaya, instead of say, Hijab. I wanted to see if this was the same for random images on the internet, and it turned out to be true.


Random lady with headscarf is more of an Abaya than the model with the Nike Sports Hijab, but both still turned out as Abaya. How Fascinating! I decided to take this further by setting up a webpage to see the other probabilities it would consider images of majority “Abaya” would be.


Following Daniel Shiffman’s examples, I checked various images and their probability percentages on a webpage I created called MobileNet vs Me.


Based on the Array results in the console, the results OTHER than abaya were significantly lower. Abaya was 66% while Cloak was at 29% and Sweatshirt was at 5%. I noticed that Cloak would be the next way in which I would be identified and then Sweatshirt. I tried this with a few other images and the results are below:


With different images, the third identifier in the array was Ski Mask and Bath Towel, which made Sweatshirt not a common third one. But yet, when I loaded the random images below:


Sweatshirt¬†came out as a second result. So we have Abaya, Cloak and Sweatshirt to identify a woman wearing a headscarf, which is very interesting to me since none of the results were specific words such as the Hijab or Headscarf. MobileNet didn’t identify them as they were known as, which is to be worn on the head for religious reasons. Perhaps a good take away from this is that the machine knows no other concept than what objects/animals/types of things that¬†the image might be similar to, which generalizes? the experience more than it specifies, regardless of how weirdly specific the identifier results were. In itself, perhaps the MobileNet dataset is Paradoxical in nature.

To step it up a notch, I decided to turn on my webcam to see if MobileNet can identify my appearance in real time, and how different the results would be as opposed to Abaya (or Cloak or Sweatshirt.)

It was interesting to investigate the capabilities of MobileNet in this manner to see if ML would elicit certain biases, of either political or sociological. What turned out was that it is quite innocent (for now) and that I may well be identified as an Abaya, a Bonnet or even a Bandaid or a Mosquito Net. These terms could even be what one would prefer to be identified with, rather than how mass media would choose to use certain terms to describe a woman with a headscarf (or anyone as near to how news outlets would describe someone who is of Islamic faith.)

To conclude, the varying probabilities that  MobileNet would give us in real time could very much be a reflection to how different even people within faith can be, from the extremes to people just simply living their lives, and should not be placed into just one universal definition.


Image Classifier Example
ML5 Homepage MobileNet Example
ml5.js: Image Classification with MobileNet by Daniel Schiffman

IFTTT – Shake, you’re Late!!!

Because announcing your late arrival to those early 8:30 classes is very important.
Because announcing your late arrival to those early 8:30 classes is very important.




Shake, I’m late!!! (no censoring needed!)

The main idea is if¬†you’re¬†running late, shake your device¬†to send a message to your colleagues on Slack! instead of typing it out as you’re running or bustling through your commute. But why Shake? I had hoped to convey this frantic state of being late expressed through shaking your mobile device, instead of having ourselves focus on the screen to type to notify as you’re walking or driving. I equated this experience to rushing/running to class, with your device either in your hand or in your pocket, shaking as you hastily move. Because with all that effort to make sure you’re not too late to class, why not let it produce a useful result? ūüôā

My goal with this experiment is to see if there can be a more dynamic experience to IFTTT than just simply pressing a button or sending a string through using p5.js. I realized if I had opened my scope to using the Adafruit feather, there would have been other possibilities, but I narrowed it down to what can be used on our mobile device, as it is always available to us. Of course, the webpage would need to be displayed first for it to work.


This applet is used to send a message to a Slack channel or a direct message to anyone on your Slack workspace through monitoring on the Adafruit IO feed. Once it reaches a certain threshold, it would send the data through IFTTT and before the user decides to turn it on, it can be customized to their own message beforehand on the applet itself. This can be used to post messages on the user’s Slack workspace, depending on the situation.



I was mostly inspired by the “Shake It” feature¬†from the LINE Chat Application. It is a popular app to use in East/Southeast Asia as opposed to Whatsapp or iMessage. I also thought about how shaking the device has only been limited for use in “sharing” or spreading contacts, and has not been used in other contexts.

Steps to adding a new contact via "Shake It" feature
Steps to adding a new contact via “Shake It” feature

I began by identifying what type of input I want to implement as a trigger to sendData(). I looked at keyPressed by perhaps creating a specific key combination for a shortcut that would send data somewhere as a notification and I also considered touchMoved for mobile devices. After looking at the examples for deviceShaken, I realized that I can really explore this for the experiment and see how it can effectively be used for an applet.

Using a default sample from the p5 deviceShaken page and creating a button to activate new channel
Using a default sample from the p5 deviceShaken page and creating a button to activate new channel.

I could have easily used a previous channel made from the example in class (the xPos or yPos, or even the message channel) but I wanted to test if merely shaking the device will create a new channel on the Adafruit IO feed itself. When that did not work, I had to initialize a new channel by creating a button on desktop, and once that was clicked, a new channel appeared by the name “colors”, since when the device is shaken, the background colors would change. Once this was done, I was able to see the values on the channel feed, confirming that the deviceShaken trigger was successful.

'colors' Channel activated
‘colors’ Channel activated
Feed after deviceShaken

The next thing I had to decide on was the “then that” part, where I searched for different search terms, such as “recommend a song” “recommend random” “tweet random” etc. My initial idea involved shaking the device and it would send me a text message or an email with a song recommendation from Spotify, but it was not an available action when connected onto the Spotify service. I had to recommend the action, and hopefully it can be realized in the future to expand the potential of IFTTT!

Available actions on the Spotify service
Available actions on the Spotify service
Applets I was Interested in
Applets I was Interested in

I narrowed my concept down to using the Slack service and decided that the shaking mechanism suits the idea best as it “quickly” notifies someone that you are running late. The Slack service also only provides 1 action under the “then that” section.

Only 1 action available under the Slack service
Only 1 action available under the Slack service

After finalizing the applet, I tested a few thresholds while monitoring the Adafruit IO feed. The first code had the values increasing by +5 every time the device shakes, which was a rigorous effort on my part to shake and check to make sure I had reached a value of 255. Since I wanted it to be quicker, I tested instances with a threshold of 200 vs 250, and changed the value from +5 every time the device shakes to +10. For this experiment I ended up with +10 with every shake to hasten the process of getting the notification on Slack. I had to test this on email first to see if the shaking mechanism can trigger to sendData, and it proved successful. The example below was triggered when the values reach 200.

Email test
Email test

When I finally tested on Slack, there was a significant delay before it posted.¬†While the email was rather instantaneous, it took around 1 minute to be able to see if it had posted. In the applet, I had set the message to post as a private message to Slackbot for the testing, and it appeared either after some time or when I “Check” the applet. This was an interesting finding as the applet would work differently for other people and also, could be affected by the amount of people using the same applet on the same Slack workspace… that would be one to test.

Testing on Slack
Testing on Slack

After testing frequently on Slack, the shaking ended up becoming instantaneous, which proves that the run time on the Applet fluctuates quite a lot. I also alternated between channel and private message to see if there were any differences, but it proved to be quite consistent.

Testing on a channel
Final test on Slack, Notification pops up as values trigger sendData()
Final test on Slack, Notification pops up as values trigger sendData()

Another aspect I wanted to explore was the visualization on the p5 webpage itself and thought to implement the example with the bouncing ball on the p5 example page with some slight color tweaks and changes. However, since it required the use of classes, I decided to go with the example from the Creation and Computation github called Color Shake, and mapped the values from the shaking action to the background color.


I believe there is still a lot of potential in creating IFTTT projects once¬†more actions in each services are added. There is still so much that can be done to enhance the way we use our devices and the applications on our phone, and IFTTT is something that could help realize that. It would also be nice to see other ways in which deviceShaken can be implemented in other projects – it’s something that is still rather gimmicky as opposed to contributing to an experience. I can see it be used in perhaps multi-sensory experiences and in games, but perhaps the sensitivity of the shaking itself is something to consider. Overall, its potential might be dependent on the limitations of our handheld devices.


p5 device shaken
p5 mapping
p5 device shaken threshold
slack applet idea
line shake it

Computational APIs – Moon Phases, When?

“When will the moon look like that? The answer is…”

Website Experience



This web page was born through my burning desire to¬†know more things about¬†The Moon and using Wolfram Alpha API to provide more information about it. I had initially wanted to dive into largest moons in the solar system (in which the earth’s moon is 5th largest) because I wanted to create an interactive version of a similar poster I have¬†on my wall. Another option was to continue within the realm of outer space and also thought of creating an information page about the Earth’s moon, which would focus on coding a fact page (Moon Phase 101), but alas, dealing with the way the answers are given through Wolfram led me to other options. As a result, I created a page to tell me when the moon phases are happening according to real time.¬†The web page became less about¬†information about the actual phases¬†themselves but more of giving the answers about “when will the moon look like that certain shape?”

My goal for this experiment was simply to how to send a query string to an API such as Wolfram’s with a single click of the image and receive¬†the feedback through text. I also wanted to pay attention to the overall look of the¬†MVP page and allow some¬†refinement on the design.



I started with using the example Nick showed in class about sending a query string to Wolfram through an input box. Since I was interested in creating a web page in the theme of the galaxy, I began by asking about the type of information that might be used. I asked questions about the different moons in the solar system and selected a few to ask as a string to Wolfram Alpha.

process-questions-02 process-questions-01


I did not expect these answers as they were specific to telling information about size, when I wanted a random fact about the moons themselves. Once I changed from using “What” to just treating the input box like a google search bar, it finally gave me a larger variety of answers. Using “tell me about” also generated a different answer, albeit long.

I realized that when working with Wolfram, you had to be clearer in what you ask for. If you use words that refer to a size, Wolfram will give you the answer relating to that. I would assume that if I wanted to ask about characteristics of the moon, I would need to have known about that information to ask it. Wolfram would not give me facts that I would not know how to ask. This contradicts the purpose of the web page, as I wanted to use it to provide more facts rather than having to search them on google before hand. Perhaps I wanted one single query that can provide different answers, but that would not be possible in this case.


Knowing this, I proceeded to change the focus to the earth’s moon. What I found amusing was when I sent the question with a “What” I received an answer that would be a response to “When.” It was only until I asked about the shape itself (“crescent”) that I would get information of the shape, which would serve the concept of “Moon Phase 101.” However, this would not work for the other shapes as Wolfram couldn’t understand the input to the names of the other phases. “What is a waning” “What is a Waxing” and “What is a full moon” would still give me the time in which the moon would be at that state in the month. I found this intriguing in terms of understanding the capabilities and limits of Wolfram Alpha. For the final page, I decided to create a page based on the answers that would imply “when” the moon will be at a certain shape, which allowed for visual¬†association of the shape to when¬†the phase occurs at the time of the month.


The p5.js aspect to this was quite simple, as I created the illusion of space in the background through randomly scattered circles that would change each time it refreshes.


I then found a free resource online on with the different moon phases and edited them accordingly.


The final challenges to this page were preloading these images and using the mouseClicked or touchStarted function that would sendtheMessage of the question string through pubnub to Wolfram.

Resizing the vector images and dividing them by 2 and arranging them accordingly on the page
The vector images arranged according to different x and y positions on the webpage. Used a reference from a previous p5 coordinate example to pinpoint all the locations.

In approaching the clicking/touch mechanism, I tried to use classes within functions to call upon the range on the images, which when activated, gives text feedback. I consulted this process with Maria and got a sense of it, but was not able to execute it. I believe in future projects or applications where it is needed, I would be able to utilize classes and perhaps create a “cleaner” javascript file.


Since it did not work, I consulted Omid and was reminded that the easiest way to complete this simple action was just to set the range within an if function. It was nice to be reminded that sometimes the simplest things are just truly, achieved through simple means.


Testing the clicking/touching of the moons and printing them on console

Each of the images would call the word to the phase itself, as shown above with “new moon.” Other phases would also send information pertaining to the shape itself, such as “waning gibbous” or “waxing crescent” and in turn, giving feedback of when it will next happen in the span of a month.


Next Steps

To further enhance the page, I would perhaps find out a way to prevent the overlapping of the response given when the images of the phases are touched. The answer is generated in the middle of the page but I would then need to find a way so that every time it is clicked, the text from the previous session disappears and is replaced with the new one. This would need experimentation with the styling of the text through readIncoming function on the basic Pubnub code and perhaps separating the strings or placing a text background if possible. I also would like to explore other types of api to enhance the page in terms of information and usage. It would be a great opportunity for me to use this theme as practice into diving deeper into css and p5.js for future web browser projects.


“9.8: Random Circles with No Overlap – P5.js Tutorial.”¬†YouTube, 15¬†Mar.¬†2016,

“Hand Drawn Polka Dot Phrases.”¬†Freepik,

“Moon Phase and Libration, 2019 Moon: NASA Science.”¬†Moon: NASA Science, 19¬†Dec.¬†2018,

“P5.js | Reference.”¬†P5.js | Home,

“Top 10 Largest Moons In The Solar System.”¬†THE LATEST SCIENCE SPACE HI-TECH NEWS, 29¬†Jan.¬†2018,

“Top 4 Keys to Understanding Moon Phases |”¬†EarthSky – Earth, Space, Human World, Tonight, 25¬†Jan.¬†2019,

XBee + Metronome – Birthday Box


Have your own birthday accompaniment with the Birthday Box!


The introduction to the XBees for our first assignment was exciting for me, as I¬†enjoy being introduced to different¬†ways to connect/network with low-fidelity setup. It felt as “magical” as using pub-nub in terms of seeing its potential used in many¬†instances that require¬†devices¬†to talk to one another.

In testing the chat through radio, I had a bit of a challenge setting up the XBee due to problems with the serial port installation. I swapped my radio kit to the Lilypad but CoolTerm still had problems recognizing my usb port. Soon after I realized that I had to install an older FTDI drive for my Mac OSX. It was a miss-step on my part (Always remember to check your operating system!) and once that was set up, the chat worked, albeit with some missing letters when it appeared on my peer’s screen. Using this as primarily a chatting tool might prove to be faulty. Regardless of the overlap when typing simultaneously with another person, it was¬†simple and¬†straightforward to use. I can see myself preferring to use this in future connectivity projects.

ATRE – to RESET configuration.
ATID – to determine the radio channel.
ATMY – to determine your radio ID.
ATDL Рto set the ID of the radio you are sending information to.


Before jumping into the output sensor for the experiment, I had to make sure that my XBee can send and receive data from other radios. We all sat together and made sure that we can accomplish connections between our radios by changing the atdl configuration. After multiple tests, I realized that there was a problem with my XBee, as it can send values but cannot receive. Even with my Arduino uploaded with the changed “PhysicalPixel” example, (adding 1 after Serial) it would not activate the LED. I tried debugging it multiple ways from resetting my settings on CoolTerm to changing the circuit and trying my XBee on different breakout boards.

What turned out to be the problem was my soldering technique. I was rather sloppy in soldering the pins to¬†my XBee breakout board and had some¬†solder¬†connecting pins next to each other. After fixing the problem, it was¬†Finally¬†able to receive values from another, with “H” turning on the LED while “L” turned it off.

Breakout board after fixing the soldering

To complete this experiment, I decided to use the speaker as I am quite familiar with it from previous projects. I wanted to look for a tune that is more familiar with others and can be used for social events. As a result, I found a tune for the Happy Birthday song and implemented that in the toneMelody code. The Arduino code for this experiment can be found here: When the XBee receives an H, the speaker will play the Happy Birthday tune.

In regards to testing it with the metronome coordinator Kate provided, the potentiometer can be used to set how frequently the song will play. Once the values are sent, the song will play until it completes and once it receives another “H”, it will then play again. The slowest duration would be it repeating every few seconds, and the longest would have a delay of about 5-10 seconds.

In forming the circuit, I had first placed the components on the breadboard but managed to place it in the box with a smaller one. A few things I had to note were to make sure that the GND on the Arduino connected to ground, otherwise the circuit would not work(!) I tested a variety of ways to place the breadboard in the box to make sure that when closed, the radio can still receive the signal. I found that by placing it nearer to the lid proved to work (as shown in the video), as opposed to just placing it in the bottom of the box. I also used a different 8 ohm speaker than the one provided the kit as it had a slightly louder volume.

Using the normal breadboard
Placing the smaller breadboard inside the box

There were also other possibilities to play around with the “H” and “L” values in terms of controlling the Happy Birthday melody. Perhaps once the “L” value is sent, the tune can stop instead of playing until the end, or as the potentiometer is turned to the slower values, the speed of the song can be altered instead of the frequency. For now at least, we have the traditional Birthday Box.