Driving with shrimp vision

RESHAWNA MAINE editorial.research@photonics.com

Mantis shrimp — a distant relative of the prawn that hang off the edges of cocktail glasses — were the inspiration for a new type of camera that could make autonomous cars even safer.

The camera, featuring a high-dynamic-range polarization imaging sensor, was developed by a team at the University of Illinois at Urbana-Champaign. Its dynamic range is about 10,000× higher than commercial cameras currently being used. By mimicking the vision of a mantis shrimp, the sensor enables the camera to better spot hazards in challenging imaging conditions, such as transitions from a dark area (a garage, for example) to a bright area (outside on a sunny day), and in hazy or foggy conditions. Compared to the color cameras currently used in cars, this new camera can detect hazards 3× farther.

Mantis shrimp (Odontodactylus scyllarus).

“In a recent crash involving a self-driving car, the car failed to detect a semitruck because [the truck’s] color and light intensity blended with that of the sky in the background,” said research team leader Viktor Gruev, an associate professor in the electrical engineering department at Illinois. “Our camera can solve this problem because its high dynamic range makes it easier to detect objects that are similar to the background, and the polarization of a truck is different than that of the sky.”

Polarization imaging can act like memory foam in that it “remembers” and encodes information such as 3D shape, surface roughness, and material or tissue structure composition. Although most vertebrate species, including humans, are not equipped with the ability to distinguish polarization of light, invertebrates have evolved with high-dynamic-range photosensitive cells to detect polarization and utilize it in visually guided behavior.

The mantis shrimp was the inspiration for a new type of camera that could make autonomous cars even safer. Courtesy of iStock/vectorfun.

Among the various vision systems within the animal kingdom, that of the mantis shrimp is most complex, with capabilities to detect 16 spectral channels, as well as four linear and two circular polarization channels. Also, individual photoreceptors have logarithmic responses to incident light intensity, which equips the creatures with high-dynamic-range imaging capabilities.

The polarization imaging sensors (polarimeters) currently used in a number of applications are outmatched by digital color cameras for frame rate, resolution, noise, and dynamic range. Such optoelectronic limitations, along with limited dynamic range, have hindered the industrial integration of polarization technology. To address them, the researchers took to the sea, imitating the complex visual system of the mantis shrimp in two ways — first, by utilizing four different pixelated polarization filters that are offset by 45°; second, by operating the sensor’s underlying photodiodes in a forward bias mode rather than the traditional reverse bias mode.

Sample image showing the co-registered color and polarization. Regular color image (a). Degree of linear polarization (DoLP) represented in a false-color map, where red and blue indicate fully polarized and unpolarized light, respectively (b). Angle of polarization (AoP) represented in a false-color map, where red and light blue indicate horizontally and vertically polarized light, respectively (c). Courtesy of The Optical Society of America (OSA).

These combined advancements enabled the researchers to create an imager that achieves a 30-fps, 140-dB dynamic range, along with a 61-dB maximum signal-to-noise ratio across 384 × 288 pixels equipped with logarithmic photodiodes.

The resulting bioinspired polarization imager is compact, with low fabrication cost, potentially making it an integral part of automotive and remote sensing applications.