Neuromorphic Sensor Recognizes Motion, Predicts Path

Facebook X LinkedIn Email
ESPOO, Finland, April 18, 2023 — A bio-inspired sensor developed by researchers at Aalto University can recognize moving objects in a single frame from a video and successfully predict where they will move to. The sensor has use in dynamic vision sensing, automatic inspection, industrial process control, robotic guidance, and autonomous driving technology, among other fields.

Current motion detection systems require a variety of components and complex algorithms performing frame-by-frame analyses, making them inefficient and energy-intensive. Inspired by the human visual system, the Aalto researchers developed a neuromorphic vision technology that integrates sensing, memory, and processing in a single device that can detect motion and predict trajectories.
Conventional sensors only capture a single moment in a frame, but the new sensor can read information about the past and use that to predict the future. Courtesy of Hongwei Tan, Aalto University
Conventional sensors capture only a single moment in a frame, but the new sensor can read information about the past and use that to predict the future. Courtesy of Hongwei Tan/Aalto University.

At the core of the technology is an array of photomemristors — electrical devices that produce electric current in response to light. The current doesn’t immediately stop when the light is switched off. Instead, it decays gradually, meaning that photomemristors can effectively “remember” whether they’ve been exposed to light recently. As a result, a sensor made from an array of photomemristors doesn’t just record instantaneous information about a scene like a camera does, but also includes a dynamic memory of the preceding instants.

“The unique property of our technology is its ability to integrate a series of optical images in one frame,” said Hongwei Tan, the research fellow who led the study. According to Tan, the information that each image contains is embedded in the images that follow as “hidden” information, so that the final frame in a video also contains information about all the previous frames.

This quality, he said, allows the researchers to detect motion earlier in the video by analyzing only the final frame using a simple artificial neural network.

To demonstrate the technology, the researchers used videos showing the letters of a word one at a time. Because all the words ended with the letter “E,” the final frame of all the videos looked similar. Conventional vision sensors couldn’t tell whether the “E” on the screen had appeared after the other letters in “APPLE” or “GRAPE.” The photomemristor array used the “hidden” information in the final frame to infer which letters had preceded it and predict what the word was with nearly 100% accuracy.
The sensor is made of an array of photomemristors. Courtesy of Hongwei Tan, Aalto University

A sensor made of an array of photomemristors. Courtesy of Hongwei Tan/Aalto University.
In another test, the researchers showed the sensor videos of a simulated person moving at three different speeds. The system recognized motion by analyzing a single frame and correctly predicted the next frames.

Accurately detecting motion and predicting where an object will be are vital for self-driving technology and intelligent transport. Autonomous vehicles need accurate predictions of how cars, bikes, pedestrians, and other objects will move in order to guide their decisions. By adding a machine learning system to the photomemristor array, the researchers showed that their integrated system can predict future motion based on in-sensor processing of an all-informative frame.

In addition, the in-frame information that the researchers attain using photomemristors avoids redundant data flows, enabling energy-efficient decision-making in real time, professor Sebastiaan van Dijken said.

The research was published in Nature Communications (

Published: April 2023
1. A generic term for detector. 2. A complete optical/mechanical/electronic system that contains some form of radiation detector.
machine vision
Machine vision, also known as computer vision or computer sight, refers to the technology that enables machines, typically computers, to interpret and understand visual information from the world, much like the human visual system. It involves the development and application of algorithms and systems that allow machines to acquire, process, analyze, and make decisions based on visual data. Key aspects of machine vision include: Image acquisition: Machine vision systems use various...
machine learning
Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on the development of algorithms and statistical models that enable computers to improve their performance on a specific task through experience or training. Instead of being explicitly programmed to perform a task, a machine learning system learns from data and examples. The primary goal of machine learning is to develop models that can generalize patterns from data and make predictions or decisions without being...
Research & TechnologyOpticsSensors & DetectorsImagingphotomemristorsAalto UniversitypredictmovementmotionAutonomous drivingsensormachine visionin-sensor processingmachine learningEurope

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.