Deep Learning Is Used to Recover Objects in Low Light

Facebook X LinkedIn Email
CAMBRIDGE, Mass., Dec. 19, 2018 — A new imaging technique, developed by engineers at Massachusetts Institute of Technology (MIT), demonstrates that deep neural networks (DNNs) can be used to illuminate transparent features such as biological tissues and cells in images taken with very little light. The researchers used a DNN to reconstruct transparent objects from images of the objects taken in near total darkness.

Deep learning technique reveals 'invisible' objects in dark, MIT.
From an original transparent etching (far right), MIT engineers produced a photograph in the dark (top left). They then attempted to reconstruct the object using first a physics-based algorithm (top right) and then a trained neural network (bottom left), before combining both the neural network with the physics-based algorithm to produce the clearest, most accurate reproduction (bottom right) of the original object. Courtesy of A. Goy, K. Arthur, S. Li, and G. Barbastathis.

To begin, the researchers consulted a database of 10,000 integrated circuits, each etched with a different pattern. Instead of etching each of the 10,000 patterns onto as many glass slides, the researchers used a phase spatial light modulator to display the pattern on a glass slide, re-creating the same optical effect that an actual etched slide would have.

The researchers pointed a camera at an aluminum frame containing the light modulator. They then used the device to reproduce each of the 10,000 patterns from the database. The researchers covered the entire experiment so it was shielded from light, and used the light modulator to rapidly rotate through each pattern, similar to a slide carousel. They took images of each transparent pattern, in very low lighting conditions (about one photon per pixel). This produced “salt-and-pepper” images that resembled little more than static on a television screen.

The team developed a DNN to identify transparent patterns from dark images, and then fed the DNN each of the 10,000 grainy photographs taken by the camera, along with their corresponding patterns.

The researchers had set their camera to take images slightly out of focus. Defocusing works by providing the DNN with some evidence, in the form of ripples in the detected light, that a transparent object may be present. Such ripples serve as a visual flag that a neural network can use to determine that an object is somewhere in an image, although hidden.

Defocusing also creates blur, which can muddy a neural network’s computations. To counter this, the researchers incorporated the physical law of light propagation into the neural network. “It’s better to include this knowledge in the model, so the neural network doesn’t waste time learning something that we already know,” said professor George Barbastathis.

After training the neural network on the 10,000 images of different integrated circuit patterns, the team created a completely new pattern, not included in the original training set. They took an image of this pattern, again in darkness, fed it into the neural network, and compared the patterns that the neural network reconstructed, both with and without the physical law embedded in the network. The researchers found that both methods reconstructed the original transparent pattern reasonably well, but the “physics-informed reconstruction” produced a sharper, more accurate image.

The team repeated their experiments with a totally new data set, consisting of more than 10,000 images of more general and varied objects, including people, places, and animals. After training, the researchers fed the neural network a completely new image, taken in the dark, of a transparent etching of a scene with gondolas docked at a pier. Again, they found that the physics-informed reconstruction produced a more accurate image of the original, compared to reproductions without the physical law embedded.

“We have shown that deep learning can reveal invisible objects in the dark,” said researcher Alexandre Goy. “This result is of practical importance for medical imaging to lower the exposure of the patient to harmful radiation, and for astronomical imaging.”

The research was published in Physical Review Letters ( 

Published: December 2018
deep learning
Deep learning is a subset of machine learning that involves the use of artificial neural networks to model and solve complex problems. The term "deep" in deep learning refers to the use of deep neural networks, which are neural networks with multiple layers (deep architectures). These networks, often called deep neural networks or deep neural architectures, have the ability to automatically learn hierarchical representations of data. Key concepts and components of deep learning include: ...
machine learning
Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on the development of algorithms and statistical models that enable computers to improve their performance on a specific task through experience or training. Instead of being explicitly programmed to perform a task, a machine learning system learns from data and examples. The primary goal of machine learning is to develop models that can generalize patterns from data and make predictions or decisions without being...
artificial intelligence
The ability of a machine to perform certain complex functions normally associated with human intelligence, such as judgment, pattern recognition, understanding, learning, planning, and problem solving.
computer vision
Computer vision enables computers to interpret and make decisions based on visual data, such as images and videos. It involves the development of algorithms, techniques, and systems that enable machines to gain an understanding of the visual world, similar to how humans perceive and interpret visual information. Key aspects and tasks within computer vision include: Image recognition: Identifying and categorizing objects, scenes, or patterns within images. This involves training...
Research & TechnologyeducationAmericasMassachusetts Institute of TechnologyMITImagingLight Sourceslow light conditionsOpticscamerasmedicalmedical imagingMicroscopydeep learningmachine learningartificial intelligencecomputer visionAIoptical imaginglight diffraction

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.