The facial appearance of actors can now be transformed during a live performance using a projector-based illumination system that tracks an actor’s facial movements and changing expressions without the need for facial markers. A key challenge of live dynamic augmentation of human faces via projection is latency — i.e., when an image is generated according to a specific pose, but displayed on a different facial configuration by the time it is projected. The Makeup Lamp, developed by a team at Disney Research, reduces latency during every step of the process from capture, through processing, to projection. The system captures a live performance in the infrared (IR) spectrum (bottom left). A target appearance (top left) is rendered and saved as albedo and offsets based on expression and spatial position (top middle). These are blended and deformed to match the facial configuration and position (bottom middle), and projected back on the performer for appearance augmentation (right). The average system latency is 9.8ms, achieved through GPU optimizations, and is further compensated for through prediction. Courtesy of Disney Research. To minimize latency, the team limited the complexity of its algorithms. It also used a coaxial camera-projection setup, where the camera that detects facial movements shares the same optical axis as the projector that illuminates the face. This enabled the team to process images in two dimensions, rather than three, and still provide consistent augmentation of the face. The optically and computationally aligned high-speed camera uses IR illumination to detect facial orientation as well as expression. The estimated expression blend shapes are mapped onto a lower dimensional space, and the facial motion and non-rigid deformation are assessed, smoothed and predicted through adaptive Kalman filtering. The desired appearance is generated by inserting precomputed offset textures according to time, global position and expression. The team evaluated the Makeup Lamp using an optimized CPU (central processing unit) and GPU (graphical processing unit) prototype and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the Makeup Lamp system fully supports dynamic facial projection mapping without the requirement of any physical tracking markers; and incorporates facial expressions. By adjusting the illumination, the system can display any color or texture and can include special effects that, for example, make an actor appear older or grow a beard, all in real-time. “We've seen astounding advances in recent years in capturing facial performances of actors and transferring those expressions to virtual characters," said vice president Markus Gross. “Leveraging these technologies to augment the appearance of live actors is the next step and could result in amazing transformations before our eyes of stage actors in theaters or other venues.” The research team plans to refine the system by further minimizing latency, reducing noise levels and by implementing more detailed tracking. “We believe that projection-based nonrigid expressive augmentation could give rise to a wide variety of creative applications in the near future,” said Anselm Grundhöfer, principal research engineer. “For instance, such a system might be used to simulate cosmetic plans on a face before actually applying makeup.” The research was presented at the European Association for Computer Graphics conference, Euographics 2017, on April 24, 2017 in Lyon, France. Makeup Lamps: Live augmentation of human faces via projection. Courtesy of Disney Research.