Special effects captured in a whole new light

Amanda D. Francoeur, amanda.francoeur@laurin.com

Imagine an imaging system that doesn’t require a camera or reflective markers but instead uses small photosensors and solid-state projectors that transmit near-infrared light to capture movement. The technique could improve the way special effects are rendered in movies, providing filmmakers with an easier, more adaptable and less expensive on-set motion capture system.

A high-speed solid-state projector labels space by emitting near-infrared patterns of light at 870 nm. Imperceptible photosensors hidden underneath a subject’s clothes receive a distinct pattern from the sequence and, by doing so, determine their own location. The projectors encase an LED and passive binary film that transmit the patterns of light.

Ramesh Raskar, an associate professor of media arts and sciences at MIT, and Dennis R. Miaw, a graduate student, created the technique they call “second skin.”

“The system is better than traditional camera-based motion capture because the markers are imperceptible and you get a unique ID per marker,” Raskar said. “Plus the actors can work in natural conditions, i.e., under studio lights and stage costumes. This makes it possible to finally achieve ‘on-set’ motion capture, a holy grail in this business.”

Lights, camera … action!

Small optical tags with photosensors are worn underneath the subject’s clothing, making the tags more discreet than conventional markers. The sensors compute location, orientation and incident ambient illumination – when the lighting matches all of the elements placed in the scene at the time the image is captured – for surface reflectance. The data is stored by a flash memory microcontroller, also hidden under clothes, that sends the information wirelessly via a radio-frequency channel to a computer.

Photosensors are embedded into a sleeve to track movement of a subject’s arm and wrist, while vibrating sensors (blue devices around the wrist) are incorporated to correct the subject’s position (left). A software program depicts the arm’s actual location (red line) and its ideal position (yellow line) (right). The dots correspond to the vibrating sensors, and the green dot indicates vibration that tells the subject how he should reposition his arm. The technique is useful for tracking and correcting motions in tai chi exercises and, in the future, it could advance physical therapy methods.

An unlimited number of tags can be integrated within the system because “the tags receive light from the projectors, and the tags themselves determine their own location based on the light sequences they receive,” Miaw said. With reflecting markers, it’s hard for cameras to differentiate each one individually, which can repeatedly result in reacquisition problems.

High-speed solid-state projectors, or beamers, include LEDs and passive binary films, or masks, set in front of them for use as transmitters. The beamers label space surrounding the tags by creating zones, and they can be mounted anywhere on set or placed outside, as long as the tags are in the beamers’ line of sight and at least 1 to 8 m away.

Every projector emits eight different near-infrared patterns at 870 nm, and the photosensors receive their own distinctive pattern, which they decode into a binary signal to indicate their position. “A photodetector that is sitting somewhere in the space will receive a unique light pattern corresponding to that 3-D point in space,” Miaw said. “If that photodetector moves to a different spot, it will receive a different unique pattern that corresponds to the new location.” At the same time, each sensor records the patterns approximately 500 times a second, and 10 projectors can acquire a frame within 800 ms.

The system uses time division multiplexing – transferring two signals within one communication channel with separate time slots of fixed length – set at 455 kHz. The beamers have their own time slot where the patterns remain on for 33 ms at a time followed by a 33-ms delay. According to Miaw, the delay is to ensure that the tags do not confuse patterns from a start bit that’s emitted at 100 ms in length at the beginning of the sequence, which may hinder synchronization.

The technique results in easier and faster production time and management, and it eliminates the need to change settings. “This system is also extremely inexpensive, with hardware costs on the order of tens to hundreds of dollars, as opposed to tens of thousands of dollars for most conventional camera-based systems,” Miaw said.

Second skin not only advances special effects for movies, but it also can be used in tai chi training and, possibly in the future, injury rehabilitation and sports training.