Group 3 – Project 1: Robot Camera

Project 1
Group 3: Wentian, Nicky, Shipra, Gavin

Final video: https://drive.google.com/file/d/1yAy_Jzsbj3LDxE-Kuq0D4yje0iWWmC43/view?usp=sharing

screen-shot-2023-02-26-at-6-55-21-pm

In project 1, our group intended to investigate the rapport between humans and robots. Rather than treating the robot arm as a clumsy machine, we personified the robot and created a live scenario where robots can interact with humans.

In the scenario we created, the robot is endowed with a “personality”, creating the illusion of it interacting with a human in a distinctive and enchanting way.

Description
While exploring the robot’s operation, one of the issues we recognized is that robots still fall into uncanny valley territory when trying to replicate human behavior. Although the futuristic robots in sci-fi movies have demonstrated possibilities of consciousness the scenario that robots are fully in possession of mortal characteristics is still not entirely plausible in today’s world. Based on the identified problem space, we wanted to experiment with the potential interactions between humans and robots personified from conscious characteristics, mindsets, and behaviors.

We brainstormed an interesting scenario where the actor interacts with the robot arm without real conversation or physical contact. To be specific, the actor will be gossiping with someone else on a phone call, and the noise she makes would wake up the robot’s arm. Then, the actor will gesture to the robot to go away, and the robot will start to pretend to not eavesdrop on her conversation until the moment that the robot hears something explosive.

Our intended outcome is that, by showcasing how robots could interact with us in daily circumstances, people (especially technological designers/researchers) could be inspired and conduct further research in the field of robot personas.

Setup
setup3   setup4   setup5

We created a setup that matched the script and storyboard. We used props such as books, plants, a water bottle, paper, laptop, and a desk lamp to create the appearance of a design studio/office. Our actor was placed directly opposite the robot arm. A hidden microphone was set up next to her to capture sound, and the phone was put in the robot arm’s gripper. We used mostly natural lighting by opening up the blinds on the window, and also aided it with desk lamp featured in the video, and an additional hidden off-screen lamp. A meta element is added to the project because the robot is looking at its own operating instructions throughout the final video.

MIMIC for Maya helped us to visualize the final video, but we decided to go with manual input for the robot arm’s waypoints, because we felt more comfortable with those controls at this point in our learning. The trickiest aspect was getting the timing correct – we had to coordinate the actor and robot to create the illusion of them responding to each other. An additional challenge was ensuring the robot didn’t solely function as a mechanical object. We gave it somewhat natural actions – looking away, glancing at their documents, being nosy – to personify it and give it some character.

We set waypoints to create the different actions of the robot. The difficult part about this was applying different speeds to the robot arm – some actions, such as the robot looking at its documents required a bit more breathing room, whereas the robot being “surprised” by what it hears needed a much faster movement. After testing this out, we used the “wait” keyframe for the part where the robot needs to pause.

 

Development Process

Storyboard and link to script
storyboardMimic images and link to Mimic output (1, 2)
screen-shot-2023-02-26-at-7-54-20-pm
After settling on an idea, we started gathering references for shots and the scripts. The intent was to be able to create an environment for the actor and the robot to be able to coexist, a scene where they could have a conversation. After writing the first draft of the script, we started framing the scenes and sequences. Since it is a short video, we broke it down into 8 major frames, and drew a storyboard based on the script. This was further developed on Mimic and helped in understanding timing, movement, and camera angles.

Post this ideation and prototyping stage, we gathered props (set elements) from the Digital Futures Lounge (6th floor) and created the environment for the shoot. Since the scene is set in a study environment, we decided to work with the robot arm and the table set up. After a few test runs with the actor, we were able to synchronize the script to the robot movements. It was easier to modify and change the dialogue delivery, as compared to changing the robot’s movement. The final shot was edited with the added effects of phone ringtone, robot arm sounds (to exaggerate the movements). The blinking action was also added to make it seem like the robot had just woken up from sleep, hearing the ringtone.

Code or Files
Ring tone: iPHONE RINGTONE SOUND EFFECT
Sound design Robot movement: Sound Design: Robotic Arm Sequence

Devs 2 – Group 5

By: Victoria Gottardi, Yifan Xu, and Firaas Khan

What are We Investigating? 

For Devs 2, we wanted to experiment with setting up some objects of different heights on the table, and try to discover how the robot arm could be capable of filming these objects to our precise measurements. We also wanted to experiment with how the gripper moves while holding the camera. For project one, we are planning on creating an advertisement, so we must be very strategic with our shots. We want to make sure the robot gives us the most affordances when creating really clean and usable shots. The positioning and movement of the phone and gripper will be crucial to getting what we’ll need.

The Process

img_20230208_125047 img_20230208_121434

Here is the link to the first attempt

Here is the link to the second attempt

 

Code

Here is the link to attempt one’s code

Here is the link to attempt two’s code

 

What We Learned

For this experiment, our main takeaway was that the gripper does not spin 360 degrees, so in order to move it in the correct position, you must make sure it is not fully turned one way to the max. We also learned that spinning the gripper too much can actually take away from the shot and potentially be visually nauseating to the viewer. In these experiments, we also played with the time of each waypoint, we discovered that the best length of time was 3-4 seconds and it appears that this amount of time could capture our desired items effectively. However, we think the length of time of each waypoint might also depend on what we are planning to show with the shot, and how we want the viewer to view and understand the shot.

 

 

Dev2- Group 2

Description of what you are investigating

As we talked about in Dev1, we have had an obvious clue since that time, we want to make an experimental film which is based on robot technique. After so many discussions, we gradually defined the concept of our film. Before the first test, we got some feedback, such as some environment setup and shooting angles, so we pay more attention this time.

In this section, we are going to know how to control the camera which can maintain our script of the story. And we also want to try how people can catch the movement of the camera and collaborate with it. So we decide people/cameras follow each other’s movement to shoot a suitable video.

 

Video

https://vimeo.com/manage/videos/795793529

 

What is it you are trying to figure out?

In the previous exercise, we moved the robot’s arm in a circular direction, and the desired subject (plate and candle) was in a fixed place.  But at this stage, we implemented the zigzag movement for the robot arm, which was in four movements to the right and left.  Also, the subjects intended for filming were moving, we also considered different speeds and stops at some points to coordinate the movement of the robot’s arm and determined the coordination between the robot’s movement and the hand’s movement.
Since the robot was not specifically designed for filming, the filming process was more challenging and time-consuming.
Therefore, our first challenge was to control and coordinate the movement of the hand and time and the movement of the robot in different directions.
Also, this project had to be filmed within the boundaries of the table with environmental restrictions, so the final sequence where the crumpled paper was thrown should not be outside the boundaries of the table and we repeated it many times to achieve the desired result.

 

Processing

  1. Drawing the step of robot movement before we shot the video.

We became more organized this time, we wrote down the script and drew the camera movement before we do the shooting. We think it is really a good way to do our plan on a robot camera. Because we should set up the movement in the programme, so we should know every move is what we want.

microsoftteams-image-2

2. Showing the paper: We decided to show the paper from far to close, which can get more feeling about the story will begin.

movement-of-robot

3. Write down some words: We set up 4 movements like Z, and the actor just follows the movement to write down the 4 words.

test01

writing

4. Close-up shot. To show the paper ball in order to let the audience know the emotion of the actor.

throwing-paper-balls

 

Describe what you learned

In this teamwork, According to the specific details of the video and filming project, we designed the movement path and waypoints. Filming with the help of a robot arm can be a unique experience because the robot moves along the specified path with high precision and control without vibration, which is difficult to achieve with human camera operators.

1. Improving camera movement: In this exercise, we were able to set smooth, accurate and repeatable camera movements, which were easier to take complex shots.  This is especially useful in film and video production where the ability to control camera movement is essential.
2. Advanced Versatility: We programmed the path programming and precise selection of waypoints to perform certain movements and move it along the filming path with precise timing, which is difficult to do with traditional camera setups.  This versatility is especially useful for shooting videos that require fast, smooth and precise camera movement. It can also save time and effort and reduce the risk of error or mistake because a human operator is not required.
Overall, the experience of using a robot for filming was very interesting for us, and this path has the potential to revolutionize the film and video production industry and offers a wide range of advantages that can improve the quality, efficiency and safety of filming.  to improve

How did it go?

In the previous exercise, we moved the robot’s arm in a circular direction, and the desired subject (plate and candle) was in a fixed place.  But at this stage, we implemented the zigzag movement for the robot arm, which was in four movements to the right and left.  Also, the subjects intended for filming were moving, we also considered different speeds and stops at some points to coordinate the movement of the robot’s arm and determined the coordination between the robot’s movement and the hand’s movement.
Since the robot was not specifically designed for filming, the filming process was more challenging and time-consuming.
Therefore, our first challenge was to control and coordinate the movement of the hand and time and the movement of the robot in different directions.
Also, this project had to be filmed within the boundaries of the table with environmental restrictions, so the final sequence where the crumpled paper was thrown should not be outside the boundaries of the table and we repeated it many times to achieve the desired result.

 

Game Of Thrones Camera Movement

got-git

 Description

After getting hands-on experience with a robotic arm in dev 1 we decided to use the Game of Thrones intro song as the primary music element for replicating a camera shot, by using robotic camera arm movements. The iconic theme music of the show has a powerful and impactful sound that adds a sense of excitement and drama to the camera movements.

camerazoom

bypass

We aim to coordinate between traditional cinematography and robot-assisted miniature, and thus creating more comprehensive and innovative ways of storytelling. We also emphasize the Incorporation of non-traditional materials as movie components and the protagonists of storytelling.

As the robot arm is a replacement for filming tools and technologies, we are trying to explore to what extent we can substitute other components in a short film clip: e.g., actors, sounds, and environmental elements.

official_opening_credits_game_of_thrones_hbo

What is it you are trying to figure out?

To get started, we would need to determine how we want the camera to move in synchronization to the music. This could involve programming the robotic arm to move in specific ways, such as moving up and down, left, and right, or in a circular motion, while creating emphasis and blurring effects at certain phrases in the song. We think synchronizing camera footage with a sound beat can create a visually appealing and rhythmically engaging experience for the viewer.

We focus on using non-traditional actors in combination with traditional filming techniques to generate familiarity and arouse emotions in the audience. In addition, we are trying to project irony through hilarious visuals and intensity through music.

Documentation of process and results

Video Link 1
Video Link 2

What you learned

The importance of exact control in terms of speed, time, and acceleration. The rhythm of the robot arm’s movement should be in accordance with background music and the actor’s entrance and exit to achieve coherence. This exact control would benefit from program software such as Maya, in order to render at the most appropriate coordination.

It is not necessary to have specific actors in the scene to maximize thematic responses/effects. There are many alternative ways of generating similar visual effects which have a lot of room for creative input. Taking overall cohesiveness and consistency into consideration is also an essential aspect which should not be ignored.

How did it go? It does not have to work to be a useful experiment.

It went well and beyond our early expectations. We brought a reflective aluminum box, and some vegetables, and pulled coloured sheets as the background. Thinking through making/experimenting is a valid design research method in terms of idea generation and exploration.

We were able to replace the background in Premier with a still image as a form of post-production, which brings in another layer of creative elements into the scene. Despite not being the focus, incorporating software into the overall production generates unique effects and expands possibilities.

 

Dev 2/ Group 3

Group 3: Wentian, Nicky, Shipra, Gavin

 

untitled-1

 

Idea

For Dev 2, we wanted to test out two specific things – the camera’s range of movement and the ability to program using Mimic within Maya. We took into consideration the visuals and images that we hoped to eventually achieve for Project 1 and made it a worthwhile experiment for Dev 2. We knew that we wanted the robot arm to zoom and pan across an actor at a desk, so we used that setup to create keyframes in Mimic for Maya, and then recorded the resulting animation. By doing so, we investigated the capacity for Mimic to help with the preproduction stage of the project, by helping us to visualize the final output of shot sizes and angle. Furthermore, we were able to see how intuitive Mimic can be for programming the robot, as opposed to the manual input that we’ve used for prior testing.

 

Process and Results

After settling the thoughts, we started exploring and programming in Mimic. In the beginning, it took us a while to learn all the fundamental terminologies and techniques so that we could have a basic understanding of how to create our intended animation. Then we built some essential setups in the Mimic such as the robot arm, table, and background. After that, we embedded the camera from Maya into the robot arm and adjusted it to the proper position. Considering our initial idea involved an actor in that scenario, we also downloaded a human model from Sketchfab and placed it in front of the camera, which is not functional but just to fit into the scene we created. Finally, we programmed the robot arm with certain movements inspired by our early ideation, and it turned out to work quite well.

 

Learnings & Outcomes

This process helped in understanding the relationship between maya, the robot arm and the scene. It was a step towards familiarizing ourselves with the tools and controls. The intent was to be able to test the shots/ scenes using Maya as a preliminary trial for Project 1. The outcomes are two videos, one displaying the scene, the actor and the robot arm holding the phone camera; The other from the camera’s point of view, displaying the shot. This helped in understanding timing, movement, and angles. This exercise was able to add value to our Project 1 prototyping process.

 

3D Model:

“SF_Girl” by Stan, February 10, 2023

https://sketchfab.com/3d-models/sf-girl-a219070284e64ddd8a51f69c207cc81

 

Video Link:

https://drive.google.com/file/d/1mQIW6Zh_D0KTh51CO6dtcA8hf1Mf6JiR/view?usp=sharing

https://drive.google.com/file/d/1saL-Pwhs0dbql0qG0pFilpfRaSnBBCj3/view?usp=sharing

 

 

 

Dev2 – Group1

Group: Dorothy Choi, Jiamin Liu, Mona Safari, Zaheen Sandhu

Description of what we are investigating:

In continuation from Dev 1, our theme is creating a simple, cinematic nature scene that involves spatial and temporal changes, such as day to night transitions and playing with the weather. This time we decided to explore a bit more with some characters for our story and the environment. We wanted to play around with light and shadow, and make use of different materials that produce shadows in an exciting way. To do this, we used everyday objects that we had handy with us, such as a blue water bottle and a flashlight on a mobile phone to give the watery effect on a grey back board.

img_1815-1 img_1816-1

What we learned / How it went:

In this group work, continuing from Dev 1, we were able to further develop details to the nature scene for filming, and movements of the robotic arm for different filming positions.

  1. Learned to manipulate different speeds of the robot arm to capture desired video effects, different frames of reference and focus points.
  2. Learned that random exploration, or just ‘trying things out’ with both the robot arm and the nature scene helps ideate and create interesting scenes to film. For example: changing the order of objects in which they are placed, or testing different focus points for lights and shadows and how it affects the scene (e.g. light from the top puts a spotlight on the characters, but the shadow of the robot arm hinders the scene), and how the robot arm can impact the filming process (e.g. producing a consistent water effect required the producers to move in sync with the speed of the robot arm).
  3. Played with light and shadow, and learned how it can impact a scene in terms of its ambiance, storytelling and potential message for the audience.

 

Our Challenges:

One of the main challenges that we faced while we were setting up our scene was having the surrounding environment in the video as well. We noticed that it was tough to be able to record the scene and not have the immediate environment exposed to the camera. To tackle this, we decided to create walls of thick sheets that would solve this purpose. While this did in fact solve our main challenge, it also acted like a bonus element for our project. We wanted to play with light and shadow and the sheet acted as a great background for that purpose.

scene1-min

Our Tests:

 

Dev1-Group 2

cover

 

2

bd937da1-aaa1-4e5a-99d4-a76cf2854039

 

Description of what you are investigating

In this mission, we study the control of the robotic arm and establish the operating range to carry out the automatic shooting. The controllability of the robotic arm has inspired us to use it in experimental video production. An experimental film is a film with a single objective language, without narration or subversion of traditional storytelling techniques. It mainly creates short films and closely relates to surrealism, expressionism, and avant-garde art. The methods used in the experimental film include defocusing, distortion, staining, repetition and quick editing of the image. Furthermore, unsynchronized sound and image, voice variations, grotesque characters and vague themes are also common styles in experimental movies.

To achieve a unique camera lens in experimental film, we think robotic arms are a good starting point for research. So we would like to test this technique on our first mission.

Video

https://vimeo.com/manage/videos/795793529

Processing

For our project, there is a frame which we will use in the edited video to show whether the candle was lit or extinguished. It is essential for this Dev1 to be familiar with the robot arm because the robot arm can complete precisely the same movement every time. Then we can have 3 similar frames, but the candles are different.

1. Turn around the empty plate. First, we marked 6 points around the plate. Then we adjusted the angle of the robot arm by hand at the first point and set it to be Waypoint 1. Besides, we moved the robot arm to each of the remaining 5 points via the touch screen to control the robot arm and turned it back to the first point. This makes the camera track around the plate 360 degrees. Thus we set up the robot arm’s trajectory and started recording the first frame – the camera circling the empty plate.

cirling-robot-1

2. Three candles on the plate. We put three candles into the plate when the robot arm was circling the second time.

put-on-candles

3. Lit three candles on the plate. We controlled the robot to stop at Waypoint2, Waypoint4 and Waypoint6 by the touch screen. Then another person can have time to light the candles.

lit-the-candles

 

 

Reference

  1. YouTube. (2006). YouTube. Retrieved February 3, 2023, from https://www.youtube.com/watch?v=PJw9az9vAf4.
  2. Matt. (2022, September 25). What is experimental film: The Essential Guide • Filmmaking Lifestyle. Filmmaking Lifestyle. Retrieved February 3, 2023, from https://filmlifestyle.com/what-is-experimental-film/

 

 

Group 1 – Devs 1 Assignment

Dev 1 assignment

Group: Dorothy Choi, Jiamin Liu, Mona Safari, Zaheen Sandhu

Description of what we are investigating:

From researching creative inspirations that we found from Behance, Instagram, and the web, we were inspired by cinematic themes in nature, miniatures, and light imagery. Our brainstorming led us to investigate how to create a simple, cinematic nature scene (e.g. animals and a cottage at the top of a hill in the countryside, near the ocean) that involves spatial and temporal changes, such as day-to-night transitions and playing with the weather (e.g. inducing fog, precipitation). We used Miro as a collaborative space to note ideas and resources that would be helpful in creating this scene. Additionally, we experimented with the Robot arm to get a sense of how it can be used to film the scene. For example, the Robot arm could be useful in holding a phone to film the scene from a panoramic aerial view, panning over the scenic transitions. We are also thinking of minimizing post-production, focusing most work on the production process.

What we learned / How it went:

In this group work, we figured out the technical principles of the robot arms from zero to one, and complete the design of how to do interaction based on the technology of the robot arms.

  1. Learned how the robotic arm works and used a robot arm to manipulate the behavioural trajectory of the robotic arm movement in the course.
  2. Searching the internet for works related to robotic arms helped us to understand the possibilities of human-computer interaction, and also helped us to open our eyes and think in a broader way.
  3. Learned from each other’s ideas in group discussions and brainstorming sessions with team members from different backgrounds.
  4. Several techniques are tested for light effects, such as using a shiny surface to make reflections on objects.
  5. For the project, we tested the possibility of changing the object’s position simultaneously with camera movements. Moving the objects and using a hand in the scene makes it feel like a theatre scene, which makes our set feel more alive!

Our Challenges:

  1. As part of the test, we used the flashlight on our cellphones to see if it was able to make a sharp contrast on the object. This was not successful on the day, so we will need to work on that for the next session.
  2. In order to make the scene look more detailed, we need to experiment with different backgrounds to see if we can create a blurry background, but the space we have is not enough to experiment with different sets, so we’ll need to figure out how to accomplish this.

Our Tests:

mindmap-devs-group-1

Resources:

Devs 1 – Group 5

Devs 1 – By: Firaas Khan, Victoria Gottardi, and Yifan Xu

What are we investigating?

For this first assignment, our main goal was to experiment with the robot arm and a camera, since our first assignment is going to be between us and the robot holding the camera. We want to investigate how the robot arm moves with a phone and phone stand attached to the gripper, what the limitations of the movement could be, other capabilities of the robot arm movement that we might not have expected, and what a short video might look like filmed by the robot arm. We figured this would be good practice for project 1, to give us a better idea of what we would be potentially working with before we got too deep into the storyboarding process for our final video.

Process and Results

img_20230125_105956 img_20230125_105958

Final Video Link

What we learned

We first built a program that had the robot clutch the iPhone while recording a brief video introducing the computer that was on the table from all sides, in order to verify the hypothesis that a smart robot is capable of spinning 360 degrees and shooting while in motion. But it didn’t turn out that way. Due to space limitations and the length of its arm, which made it simple for things to collide with it, the robot was unable to complete the 360-degree rotation assignment. At that point, we noticed that when we built up the project, the space and the robot’s structure had not been taken into account. We updated our software to a successful half-circle spin after carefully observing and measuring the area and the robot’s construction. To sum up, through our initial interaction with the robot, we have discovered that the first thing we need to keep in mind when beginning a new program is the robot’s entire environment.

Link to Code

 

 

 

Dev 1- Group 4

Team Members: Anas Raza; Chitra Chugh; Ellie Huang; Ricardo Quiza Suarez

Description of what you are investigating (100 – 200 words)

For the first experiment, our team is investigating different perspectives of cinematography and their effects on the rendering of scene dynamics. Each of the team members has presented an idea and brought materials for testing. Anas decided to use the robotic arm for light painting, specifically drawing text using long camera exposure capturing a robotic arm holding a light source. Ellie is trying to explore close-up shots to top view shots, recreating a miniature landscape with elements of abstract sculptural objects. Ricardo delves into the question of how we can translate pop culture elements, such as memes or iconic video game scenes, into a precise perfect camera shot with the aid of a robot? And Chitra is trying to capture the speed and motion using the anime character toys widely known as beyblades. She is trying to create an intra-interactive  pattern between the objects and then using a robot arm to capture the zestful dynamics of the toys. She is trying to achieve a comical violence essence through this idea.

What is it you are trying to figure out?

For dev 1, we are trying to experiment with a diverse range of possibilities with robot cameras, in terms of rotation, movement, control and timing. We are striving to familiarize ourselves with the robot arms and the technicality of it in order to better achieve optimal collaboration, specifically how the robot can aid in creating a precise scene. In addition to the technical part, we are exploring multiple themes incorporated into the cinematography of robot cameras and how unique effects can be generated via such methods.

Documentation of process and results

Anas:

screen-shot-2023-02-03-at-10-34-15-pm screen-shot-2023-02-03-at-10-34-07-pm

Ellie: https://youtu.be/IIKlochGF_M 

Ricardo:https://www.youtube.com/watch?v=kTqOWqnFJNg&feature=youtu.be

The scene: Boo haunting a character (Mario usually) until it touches him and Mario dies.

The props:1 Boo figure + another figure. Human hands to assist movement.

Death by Boo example links (what we are trying to replicate):

https://www.youtube.com/watch?v=G9szadadkpM

Boo movement pattern:

https://youtube.com/clip/Ugkx7IcOfZwYgVgBA1ZQEQQ7d_f-ZI5UNuXS

Sketch diagram for setting up the scene:

screen-shot-2023-02-03-at-10-32-16-pm screen-shot-2023-02-03-at-10-32-06-pm

Description of what you learned (100 – 200 words)

  • Importance of speed and acceleration: altering a different speed and acceleration completely changes the way of storytelling from the robot camera. There are a lot more that could be further explored in terms of the accurate control of robot arms.
  • The project has to be thought about with the limitations of the setting. That includes, the length in a grounded setup can not be larger than the table, and the robot itself has many limitations regarding how much it can displace its parts. 

How did it go? It doesn’t have to work to be useful experiment. Give a brief description of what you learned (100 – 200 words) 

  • Human components: human’s appearance adds on another layer of playfulness and dynamics to the whole scene. On one hand, hiding human interaction makes the whole cinematography intriguing when the objects “come alive”, while on the other hand, having human components in the scene registers human presence and the collaboration between robot and human is strengthened. 
  • Learning on team collaboration: when one person is maneuvering, others could be helping with holding the bottom, checking from different angles and et cetera. A good and smooth collaboration maximized efficiency.
  • Regarding robot arm placement and sturdiness, we discovered that the arm could not move more than a speed of 250mm/s. The safety feature was halting the arm when we tried to move the arm more than 250mm/s. This is because of the mechanical vibration that was produced as a result of high speed and unstable base.