In thinking of this idea, I was wondering how could I make it so that the person, when only using their nose as a controller, can start and stop the drawing action so that it doesn’t end up being a continuous line drawing all the time?
Solution 1 : Maybe I could track the distance of the nose from the webcam so that whenever the person leans forward or backwards the drawing action would turn on and off.
Solution 2: Another idea would be to use a different part of the body as to toggle the drawing action.
Creatability though it makes working with PoseNet much simpler, is restrictive in that in simplifying the process, only allows you to track one body point at a time. I will be returning to the poseNet with ml5 tutorial which does the same thing but with less abstraction.
One thing I like about creditability is that since a lot of code is abstracted, you can refer to parts using strings such as ‘nose’, ‘rightEye’ instead of having to remember indexes in the pose array as in with PoseNet. The library would work well in instances where you need only to track a single part of the body.
Trying to use leaning back and forth to change the background of an image. Below are the results of my explorations. To determine the distance from the screen, I calculated the distance between my eye and my nose as when a face is closer to the screen, the distance between the two is larger and smaller when the person is leaning back. I then hid the video and tested leaning back and forth to change the background of my sketch as seen in the green and red. I set a range whereby when the distance was less than 25 or greater than 70 and I set an isDrawing boolean value to false.
I was not able to create the drawing lines with my nose effect because for some reason the drawing action is flipped so that there is a mirroring effect which makes it confusing when trying to think and draw. Additionally, I couldn’t figure out how to make a continuous line instead of dots. When tracking the mouse I would have been able to mouseX, mouseY and the pmouseX, pmouseY positions but I wasn’t able to do that with tracking my nose. I believe this would be possible if created an array and stored all the points where my nose had been, then used the draw function to draw those points instead. I decided to continue exploring other things. Below is a screenshot from my attempts at drawing with my nose.
Observations: Using leaning as an interaction turned out not to be so great. It was hard to control the reaction to the movements because you either had to be still and lean at the same time. Additionally, it just didn’t feel too natural, especially for drawing. Perhaps a different body part would have been better although I feel this just adds more complexity to the interaction.
Reading through past projects and works:
Text Rain (1999) by Camille Utterback is an art installation where one’s body is tracked and used to interact with digital text. This piece uses color black or white to detect where the body is and then animates the text accordingly. I think it would be interesting to see if I can achieve something similar with the body tracking technologies we have today. It might be harder to achieve with PoseNet as it doesn’t give you the skeleton region perhaps with the Kinect. The animation below is an example from the PoseNet sketchbook that tracks a point on the body and tracks it as a person dances then draws text animated on that path. I will look into this further in our interaction explorations for week 5 &6
Technology as a tool vs medium?
In my explorations this week I keep coming across artists and technologists who are asking “what can the technology do for me” and the idea of this body-tracking tech as a tool to convey meaning. e.g. “We quickly discovered that PoseNet was only interesting to Bill if it helped him convey meaning. The tech wasn’t an end in itself, it was only useful to him as a tool for artistic expression.” I also came across the same sentiment on Utterback’s website and it is something I will be reading more up on this week. Utterback links to this chapter as a reference to her Text Rain project on her site. The Tool Model: Augmenting the Expressive Power of the Hand – Inventing the Medium – Principles of Interaction Design as a Cultural Practise. “A lesson from the design of the google experiment was that the technology itself shouldn’t be the star of the show…push the boundaries of digital interaction design beyond the current standard of clicks, presses, and taps” – Maya Man
Week 1 – 2: Literature and tech review & exploration of related works and related projects.
Objective: Discover what is possible with real-time pose tracking, learn about what technologies and devices can be used and how they can be used, put together a reference list/document of our discoveries, understand real-time pose tracking better. Discover more diverse, BIPOC creators working in the field and explore critical discourse.
Applications Beyond Identifying Things In Our Physical World (Visual Descriptors of Humankind)
This article describes Stanford University’s Vision Lab projects and plans for AI research, at the time of the article the lab was working on creating AI first responders to help save lives during crisis. Computer vision faces many challenges because vision is humankind’s most complicated cognitive ability and it is critical to how we understand the world.
“Today, computers can spot a cat or tell us the make, model, and year of a car in a photo, but they’re still a long way from seeing and reasoning like humans and understanding context, not just content. (A bat on a youth baseball field and at a crime scene has two very different meanings.)”
We are still a long way from computers being seeing and reasoning like humans and understanding context, not just content.
“Understanding vision and building visual systems is really understanding intelligence…And by see, I mean to understand, not just to record pixels.”
Computer vision isn’t just about identifying things in our physical environment but can actually reveal details and provide insights on things that we don’t even know yet:
Every day, the Internet generates what Li calls the “dark matter of the digital age”—trillions of images and videos and other scraps of digital minutiae. More than 85 percent of content on the Web is multimedia imagery—and it’s a chaotic mess. “There is a fundamental reason that we need to understand this,” she says. “The recording of our lives, our daily activities, our relationships—be it my personal life or what’s going on in society—is in these contents.”
Computer vision can have various applications such as monitoring and combating the effects of climate change, building smart homes, and aiding in the medical field but it can also be used for nefarious purposes such as intrusive surveillance. An important factor in combating this is diversity in technology research and AI which will ensure checks and balances when creating AI.
“Every technology can be an enabler of vices,” she says, “but as a scientist you have to have that social awareness and be very aware of these potential risks.”
Facial Recognition Software is Bad at Recognizing Faces
This article describes MIT research that found that computer vision was bad at facial recognition when it came to black women, failing ⅓ times when compared to being accurate 99% of the time when subjects were lighter skinned males.
Machine learning algorithms can discriminate based on classes like race and gender because of the lack of diversity in their datasets that were used to train the AI which are composed of about 79-87% of lighter-skinned subjects. Limited data sets can impact the effectiveness of artificial intelligence, which might in turn heighten bias against individuals as AI becomes more widespread.
We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%).
The maximum error rate for lighter-skinned males is 0.8%.The Substantial disparities in the accuracy of classifying darker females, lighter females,darker males, and lighter males in gender classification systems require urgent attention if commercial companies are to build genuinely fair, transparent and accountable facial analysis algorithms.
What role will AI play in the future of our systems? At the moment AI-based systems aren’t being used in making high-stakes decisions such as determining a person’s prison sentence, but are being used to identify subjects against hot lists of people suspected of gang activity or of having open warrants, so what does that mean when the algorithms are biased and faulty?
Cornell University: Intro to Computer Vision Class
Transmedia artist and associate professor, who creates platforms for dialog about artificial intelligence (AI) as it intersects race, gender, aging, and our future histories. Her work calls to question the need for diversity in creation of AI as artificial intelligence is already permeating our daily lives in fields such as criminal justice, healthcare, education, and employment were it affects diverse communities in different ways.
Addresses: questions of bias in AI, consciousness, data sovereignty and social equity.
Project of Note:
Not The Only One (N’TOO): is a voice interactive AI, that acts as a storyteller telling a multigenerational tale of one black American family. The bot is trained on conversations and interviews between three generations of Dinkin’s family – her, her aunt, and her niece and is an amalgamation of their experiences, histories, and unique life stories.
Is a visual artist whose work looks at the intersection space, place, and cultural memory. Her work draws on her experience as an American of mixed German and Japanese descent, living between Germany, Japan, and the US.
Project of Note:
Land of Cloud: a virtual reality and 3D sound installation (2017-2018), which I thought was an eerie yet beautiful way to question our obsessive use and over dependence on technology, where it seems like we worship our tech especially smart phones.
“Three days journey beyond Space and Time lies the Land of Cloud. The people there are silent. They communicate not through speech, gesture or gaze, but instead through strange and wondrous “cloud mirrors.” These devices keep them in constant contact with their deity, The Cloud, in whose image they are created – their bodies are themselves composed of softly billowing clouds.
The Land of Cloud is a beautiful garden, but the Cloud People are oblivious to their surroundings. They stare into their devices, motionless, spellbound by whispers from The Cloud. The garden slowly envelops them in its boughs.
If you visit the Land of Cloud, you will hear a susurration of voices when you enter the space. If you place your head inside the head of a cloud person, walking up to them, sitting or lying down next to them, you will hear that each one repeats its own mantra, given by the Cloud Deity.”
Note: Thiel provides a list of VR/AR theory and suggested readings on her site that have informed her own practise which may be interesting to look at.
Hyphen Labs is a team of international women who combine art and technology to protect, represent, and honour women of color. source I thought their Neurospeculative Afrofeminism project was an interesting way of using virtual reality as radical self-care, especially when a lot of women of color never take time for, or feel like they cannot take time for their own self-care.
“Enter Brooks’ Salon, a beauty salon of the future, and peruse a line of speculative low- to high-tech beauty products while waiting for your appointment to get a set of “Octavia Electrodes,” transcranial extensions designed to make the brain’s synapses more excitable and primed to increase neuroplasticity.” source.
Note: Torontonian & OCAD grad, creative technologist Michelle Cortese worked on this project. Her work blurs the lines between artist and engineer and encompasses physical computing, art direction, fabrication, and graphics programming. source
Indiewire write-up on the project – details the installation experience well.
Intro to Processing & Kinect by Daniel Shiffman
We used the Kinect version one and connected it through the kinect library in processing. Below are some experiments we conducted using the Daniel Shiffman’s processing and kinect videos and examples from the library. I learned that the Kinect v1 has an in-out system where an infrared light is emitted of one camera and when it reflects off a surface is read back by an RGB camera on the device which can detect colours. Kinect v1 may give inaccurate readings on the depth of a pixel because the depth camera and the RGB camera are separate so they readings as skewed, this was fixed in v2 by combining the 2 sensors. For our explorations we mostly focused on the point cloud example and were particularly interested in testing whether Kinect could detect through windows to see a person’s surroundings. We found that it could not and deduced that the transparent glass allowed the IR light to pass through and may have refracted it, additionally nearby building were still too far away for the Kinect to detect any bouncing back IR light if any. Below is a video of our window test…the areas where the screen goes black is when we were attempting to “look” outside the window.
We also found that the Kinect was able to detect better when camera was pointed against flat surfaces as seen in the second test below.
We also tested Snapchat’s new gender swap filters and concluded to never trust another digital photograph again! LOL. The filter has also raised some criticism from the trans community for transphobia
The filters have been used by many in the cis-gender community as fun and gimmicky and have been viewed with criticism by those in the trans-community especially when they are accompanied by derogatory comments mostly from straight men e.g in this twitter post that swapped the genders of famous footballers. Views are conflicted as others have viewed it as something that makes a joke out of transitioning while others think it has actually helped those who have been thinking of transitioning to see what is possible.
Computer Vision (creative) in Africa
I wanted to explore more about computer vision and graphics use in Africa in the creative sector. It was difficult to come across many artists however I came across several game designers, fashion designers, and filmmakers who were using virtual reality. I learned that in Africa virtual reality is mostly used for storytelling and particularly with the themes of African mythology and legends. VR provides a sense of agency for African creatives who through their work can challenge dominant narratives of Africa. I also learned that at the moment in Africa, South Africa is leading in the use of virtual reality and this momentum was fuelled by the corporate sector driven by the use of AR and VR in advertising. There is also a barrier to access due to cost of VR headsets so the technology is still out of reach for most people at the consumer level. I will be continuing to look at the use of computer vision and graphics in Africa particularly as I am interested in the storytelling aspects of the technology. The following links are some of the resources I will be looking at further in the coming weeks…updates will be added to this post.
Cool things I’m continuing to check out:
Celebration of Black History Month in virtual reality by Oculus.