The “Pay attention” bot

The “Pay attention” bot helps the user to keep in tune with the “real world” around them, even if they’re really absorbed into their work or listening to loud music. The bot “pays attention” for you, listening for someone to call your attention. Once it detects someone trying to call your name, it will wave at you incessantly until you acknowledge it and turn it off. By now, you are well aware that someone in the real world was looking to get your attention.


Code available on Github.


experiment-3_bb


The process for this project was rather quick and uneventful, unfortunately.

My first idea was to have the arduino itself record and process the speech recognition with the desktop printing out alerts but two things stopped me:

  1. I wasn’t very interested in purchasing a new board on such short notice, in case it doesn’t work out; it’s a pretty big commitment!
  2. I don’t like desktop or push notifications. They vex me.

So I decided to reverse the role. Have the computer, which already has microphone access record and process speech, and have the arduino nag me when something gets recognized. The next task I had was to find a suitable library or API that helped me with voice recognition. P5 was the first to offer one up. At first, I was skeptical of it, since it seemed really lightweight, so I started looking at alternatives. IBM Watson’s API seemed really interesting, but they weren’t offering it for free. There were some alternatives I could have used such as interfacing with Watson through Pubnub, but the interface of Pubnub seemed convoluted at best, and pay-to-play at worse. As someone who’s very used to getting their hands elbow-deep into code,  using an interface to do the work for me was both a very disorienting and very frustrating experience. I decided to drop this route altogether.

I went back to investigating P5’s speech and speechrec addons. For my purposes, I needed it to record continuously. There is a continuous option available, but the example online wasn’t working and I couldn’t get it to work myself, either. I distinctly remember reading somewhere, in a release statement probably, that the continuous function was buggy and to use something else instead, but I can’t, for the life of me, find it anymore. I should have took a screenshot. I’m still not used to having to document my process when I’m coding and debugging. I’ll remember to next time.

Anyway, I ended up finding a workaround. Simply assigning an OnEnd() function to the recording object and asking it to restart itself was sufficient enough for my needs. There was a small issue in testing where it would stop recording (evidently) in the time it takes for the state to change from “ended” to “started”, so it wouldn’t detect sounds for that small period of time. Given more time, I would have tried harder to get the Continuous option to work, but I have learned not to linger on the small things when you need a deliverable in a short amount of time.

Hmmmm
Debugging the “restart if you stop” function.

After I got that part of my P5 code working, it was very simple to activate the servo through serial input. There was a tiny hiccup when I was doing

if (myRec.resultString == "Roxanne") {...}

which wouldn’t pick up my name if it was stringed inside a sentence. For example, something like “Roxanne, do you have a minute?” would be ignored.  I converted the code to

if (myRec.resultString.search("Roxanne")) {...}

in order to search the string for my name, instead, and it worked beautifully. The if statement prompted the serial port to send through a code which my arduino was listening for.

In practice, the speech recognition was not as powerful as I would have liked (saying “Roxanne” often produced the words “Rock band”, instead), but it was sufficient for a prototype.

The arduino code was fairly trivial since its job was also fairly trivial. It was a slightly modified version of Kate and Nick’s basic servo code. I simply added a clause for a button press, which would toggle off the variable “shouldBeMoving” as well as a check for incoming serial data. If there was serial data, and it was the code I was feeding from P5, then I would toggle on my “shouldBeMoving” variable in order to activate the basic servo code. The arm on the servo was programmed to simply wave in a 90 degree angle, enough to be annoying and catch my attention, but not enough to be obnoxious to others.

so much code...
code code code…

I believe I broke the servo when attempting to graft the arm onto it, since it was working without fail before the arm. After the arm, the servo appeared to get tired or simply get stuck on itself after a few swings. It ended up making the button somewhat extraneous, since it was stopping itself, but I kept the button, simply because there is something satisfying about smashing the button to stop the servo, but also, just in case the servo was feeling exceptionally peppy and decided to keep waving for eternity.

servo!
Thanks Sean for the help in making the arm!

Another small thing I found frustrating but, you know, was kind of necessary, was debugging. The P5 speechrec addon requires a server to function, so I needed to have my code re-uploaded to my github page whenever I wanted to test a change. Debugging became especially frustrating since that’s usually a process where I add logging statements at different places in order to glean information, and then remove promptly when expected outcome happens. This made for quite a lot of commits, which the github page was slow to catch. I’d often have to wait about a minute or two between commits before my page would be up to date. But alas, such is the way.

so much commit
If you judge my commit messages… I really cant blame you…

Video of it working:

It lives! from Roxanne Henry on Vimeo.


References and thanks:

Servo test code

P5 Speech

IBM’s Watson

Kate and Nick’s servo code

Sean Harkins for help with making the arm

Dave Foster for help with making the box

Leave a Reply