top of page

Projects | Acoustruments

Acoustruments is a project by Disney Research Pittsburgh that was demoed at SIGGRAPH 2015. The research presents a method of creating "low-cost, passive, and powerless mechanisms, made from plastic, that can bring rich, tangible functionality to handheld devices". The technology uses the ultrasonic sensing capabilities of handheld devices, machine learning, and 3D design primitives in order to make these external applications. (For a full description and the official research page, click here.)

 

As a summer lab associate for Disney Research, I worked on the three demos that would showcase this technology at SIGGRAPH 2015. These demos included an interactive toy that would house an iPhone, an iPad game that utilized different Acoustruments mechanisms, and a Google Cardboard experience that used Acoustruments controllers. The team that produced these demos included Moshe Mahler, Eric Brockmeyer, John Mars, Gierard Laput, and me. I was mainly responsible for conceptualizing a story about the technology, producing concept art for the visuals in each product, modeling the interactive toy, producing assets for the VR experience, and implementing the iOS game.

 

Awkbot, the Interactive Toy

The narrative experience of Acoustruments starts with Awkbot, the interactive toy robot. Essentially his head space would be an iPhone screen, so I had the challenge of designing a character that accommodates an iPhone 4 and has space for the interactive controllers.

 

Below you can see the several design iterations that I went in order to flesh out the model, the facial reactions, and the interactive joints. After we chose a design for the model, I programmed the state changes in the iPhone and created the face animations in After Effects. Once the model was 3D printed and fitted with the iPhone, we used machine learning to train the toy to recognize the different states:

 

  1. Arms down = Normal, white

  2. Arms up = Happy, green

  3. Button pressed = Angry, red

 

The toy ended up being our most well-received demo at SIGGRAPH and brought a lot of laughs. I hope to see this technology used in modern day toys.

 

Factory, the iPad Game

Awkbot's story continues with Factory, the iPad Game. In this game, you can see how Awkbots are created. In order to produce an Awkbot and win the game, the player must input the correct physical mechanism and control the factory components along 5 stages. The stages include:

 

  1. The Mold Press, activated by the pressure sensor

  2. The Color Selector, activated by the rotary encoder

  3. The Painter, activated by the valve

  4. The Electrifier, activated by the rocker switch

 

When all stages are complete, you get a victory message and produce an Awkbot of the color that you selected.

 

I was most involved in this application, as I modeled and composited the assets with the help of Moshe Mahler, animated the components in Maya and After Effects, and implemented the game in iOS + Swift. Below you can see the concept art stages and then the final animation of the iPad game.

Google Cardboard

Awkbot's story ends with the Google Cardboard virtual reality experience. In this VR game, the player can navigate Awkbot's room and interact with different objects in the scene. We used Acoustruments technology to supplement the Google Cardboard experience with controllers on the side of the headset. This gave the gameplay more functionality, and with just three side buttons we could map up to 7 different interactions. I was responsible for modeling the game assets.

 

The game itself did not make it in time for the SIGGRAPH Emerging Technologies Conference, but you can still see the concept art below.

bottom of page