Deep Learning Enables Structured Light 3D Polarimetric Imaging

Facebook X LinkedIn Email
NANJING, China, June 13, 2022 — Researchers at Nanjing University of Science and Technology demonstrated a dual-frequency multiplexing fringe projection profilometry (FPP) technique that is enabled by deep learning. The researchers said their approach to FPP, which is a noncontact measurement technique for 3D imaging, enabled single-shot, unambiguous, high-precision, structured light 3D imaging.

As part of the demonstration, the researchers showed a deep neural network that was trained to directly recover the absolute phase from a unique fringe image that involves spatially multiplexed fringe patterns of different frequencies. They performed experiments on both static and dynamic scenes, and they showed that the approach is robust to object motion and could be used to obtain high-quality 3D reconstructions of isolated objects within a single fringe image.

The researchers’ technique specifically supports the ability to perform 3D shape reconstruction, filling a so-called gap that exists between 3D imaging and 2D sensing to deliver high-precision 3D image reconstruction using only one pattern as part of a structured light 3D imaging system. In FPP, a projector projects a series of fringe patterns onto a target, and the camera captures images that are modulated and deformed by the object. From the captured fringe patterns, phase information of the object can be algorithmically extracted via Fourier transform and phase shifting methods, for example.

FPP is widely used in optical metrology due to its noncontact, high-resolution, high-speed, and full-field measurement capabilities. These features are advantageous in intelligent manufacturing, reverse engineering, industrial inspection, and heritage preservation.

The researchers constructed two parallel U-shaped networks with the same structure. One network used a dual-frequency composite fringe image as the network input, which is combined with the traditional phase-shifting physical model. This combination is used to predict the sine and cosine terms to calculate the wrapped phase map.

The researchers designed the other network to predict the fringe order information from the input dual-frequency composite fringe image.

A neural network trained on data sets was then used to “de-multiplex” high-resolution, spectrum-crosstalk-free phases from the composite fringe to reconstruct a high-accuracy absolute phase map that supports the imaging process.

Traditional multifrequency composite methods cannot guarantee single-frame high-accuracy 3D imaging, and the researchers said that the work addresses this bottleneck. Additionally, it opens opportunities for single-shot, instantaneous 3D shape measurement of discontinuous and/or mutually isolated objects in fast motion.

The team plans to explore more advanced network structures and integrate more suitable physical models into deep learning networks.

The work was supported by the National Natural Science Foundation of China and received additional funding.

The research was published in Opto-Electronic Advances (

Published: June 2022
The processes in which luminous energy incident on the eye is perceived and evaluated.
machine vision
Interpretation of an image of an object or scene through the use of optical noncontact sensing mechanisms for the purpose of obtaining information and/or controlling machines or processes.
Measurement of surface roughness or quality through the use of a diamond-pointed stylus connected to a coil in an electric field. As the stylus is traced across the surface, a current is created that corresponds to the roughness.
The science of measurement, particularly of lengths and angles.
neural network
A computing paradigm that attempts to process information in a manner similar to that of the brain; it differs from artificial intelligence in that it relies not on pre-programming but on the acquisition and evolution of interconnections between nodes. These computational models have shown extensive usage in applications that involve pattern recognition as well as machine learning as the interconnections between nodes continue to compute updated values from previous inputs.
deep learning
Deep learning is a subset of machine learning that involves the use of artificial neural networks to model and solve complex problems. The term "deep" in deep learning refers to the use of deep neural networks, which are neural networks with multiple layers (deep architectures). These networks, often called deep neural networks or deep neural architectures, have the ability to automatically learn hierarchical representations of data. Key concepts and components of deep learning include: ...
In a periodic function or wave, the segment of the period that has elapsed, measured from some fixed origin. If the time for one period is expressed as 360° along a time axis, the phase position is called the phase angle.
fourier transform
Any of the various methods of decomposing a signal into a set of coefficients of orthogonal waveforms (trigonometric functions).
structured light
The projection of a plane or grid pattern of light onto an object. It can be used for the determination of three-dimensional characteristics of the object from the observed deflections that result.
The combination of two or more signals for transmission along a single wire, path or carrier. In most optical communication systems this is referred to as wavelength division multiplexing, in which the combination of different signals for transmission are imbedded in multiple wavelengths over a single optical channel. The optical channel is a fiber optic cable or any other standard optical waveguide.
visionmachine visionprofilometryfringe projection profilometrymetrology3D measurementimage reconstructionneural networkdeep learning3D imagingmanufacturingindustrialphasesmart phase mappingFourier transformResearch & TechnologyeducationNanjing University of Science and Technology (NJUST)structured lightmultiplexing

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.