Search
Menu

The Relightables System Captures Character Lighting for Virtually Any Environment

Facebook X LinkedIn Email
MOUNTAIN VIEW, Calif., Nov. 21, 2019 — Computer scientists at Google have developed a system for high-quality, relightable performance capture. The volumetric capture system, called The Relightables, can capture full-body reflectance of 3D human performances and seamlessly blend them into a new environment through augmented reality (AR) or into digital scenes in films and games. Character lighting can be customized in real time.

The team designed a novel active depth sensor to capture 12.4-MP depth maps. It then designed a hybrid geometric and machine learning reconstruction pipeline to process the high-resolution input and produce a volumetric video. The researchers then generated temporally consistent reflectance maps for dynamic performers by leveraging the information contained in two alternating color gradient illumination images acquired at 60 Hz.

The Relightables volumetric capture technology from Google.

Computer scientists at Google have created a new, comprehensive system that is able to capture full-body reflectance of 3D human performances and seamlessly blend them into the real world through AR or into digital scenes in films, games, and more. Courtesy of SIGGRAPH Asia.

The Google team demonstrated The Relightables on subjects that were recorded inside a custom geodesic sphere that was outfitted with 331 custom color LED lights, an array of high-resolution cameras, and a set of custom high-resolution depth sensors. The system was able to capture about 65 GB per second of raw data from nearly 100 cameras. Its computational framework enabled data to be effectively processed at this scale.

The Relightables system can capture the reflectance information on a person — that is, the way lighting interacts with the person’s skin. It can record people while they move freely within the volume, making it possible to relight their animation in arbitrary environments. The system combines the ability to realistically relight humans for arbitrary environments with the benefits of free-viewpoint volumetric capture and new levels of geometric accuracy for more dynamic performances, the researchers said.


Historically, cameras have recorded people from a single viewpoint and lighting condition. The Relightables allows the user to record a person and then view the person from any viewpoint and lighting condition, eliminating the need for a green screen to create special effects, and allowing for more flexible lighting conditions.

The interactions of space, light, and shadow between the performer and the environment are critical to creating a sense of presence. Beyond just cutting and pasting a 3D video capture, The Relightables system gives the user the ability to record a person and then seamlessly place that person into new environments — whether in their own space for AR experiences, or in a virtual reality experience, film, or game.

The Relightables system could significantly improve the level of realism achieved when volumetrically captured human performances are placed into arbitrary computer-generated scenes. The Google team planned to present the components of The Relightables system, from capture to processing to display, at ACM SIGGRAPH Asia, held Nov. 17-20, 2019, in Brisbane, Australia.

The research was published in ACM Transactions on Graphics (www.10.1145/3355089.3356571). 



The Relightables: volumetric performance capture of humans with realistic relighting. Courtesy of K. Guo et al.

Published: November 2019
Glossary
augmented reality
Augmented reality (AR) is a technology that integrates digital information, typically in the form of computer-generated graphics, images, or data, with the real-world environment in real-time. AR enhances the user's perception of the physical world by overlaying or combining digital content onto the user's view of the real world, often through devices like smartphones, tablets, smart glasses, or specialized AR headsets. Key features and principles of augmented reality: Real-time...
virtual reality
Virtual reality (VR) is a computer-generated simulation of a three-dimensional environment or experience that can be interacted with and explored by an individual using electronic devices, such as a headset with a display. VR aims to create a sense of presence, immersing users in a computer-generated world that can be entirely fictional or a replication of the real world. It often involves the use of specialized hardware and software to provide a fully immersive and interactive experience. ...
Research & TechnologyBusinessAmericasLEDsLight SourcescamerasImagingGoogleThe Relightablesaugmented realityvirtual realityAR/VRConsumerEntertainmentVideolightingvolumetric capture

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.