Whole-Brain Big Data Processing Enables VR via Optogenetic Control

The massive amount of data processing required to image whole-brain activity makes real-time analysis and closed-loop research in neuronal dynamics extremely challenging. Yet, real-time analysis of large-scale brain imaging is crucial to understanding how the brain functions.

Inspired by the data processing techniques used in astronomy, researchers at the Chinese Academy of Sciences developed a big data processing system for neuronal activity. Known as the FX system, it operates in real-time to analyze large-scale, whole-brain neuron activity and facilitate the closed-loop study of brain functions. In addition to supporting closed-loop neuroscience research, the FX system enables whole-brain, optical interface-mediated virtual reality.

The FX system is based on a field programmable gate array graphics processing unit (FPGA-GPU) hybrid architecture. The researchers exploited the flexibility of FPGA programming to create an optical neural signal preprocessing approach that regularizes signals from optical sensors. These signals are sent to a real-time, GPU-based processing system that performs high-speed nonlinear registration and extraction and decoding of neural signals. The real-time processing system also obtains feedback signals for controlling external devices.

The researchers adapted the FX system to the whole-brain imaging of awake larval zebrafish. They used the system to continuously monitor neuronal activity in whole brains of zebrafish and were able to generate feedback signals with a feedback delay of less than 70.5 ms.

Using the FX system, the researchers performed real-time analysis of hundreds of thousands of neurons in the zebrafish brain. They decoded the activity of arbitrarily selected neuron ensembles in order to control external devices.

The FX system can perform real-time registration, signal extraction, and analysis on data streams of up to 500 megabytes per second (500 MB/s). It can extract activity from up to 100,000 neurons. It enables closed-loop study of neural dynamics.

The researchers used three examples of closed-loop neuroscience research to demonstrate the system’s capabilities.

In the first example, the researchers linked real-time optogenetic stimulation to the activity of arbitrarily selected neuron ensembles. They functionally clustered neurons in the whole brain into ensembles and used the spontaneous activity of selected ensembles as a trigger signal to implement real-time optogenetic stimulation on targeted neuron ensembles. The closed-loop stimulation effectively activated downstream brain areas, compared to open-loop stimulation.

In a second example, the researchers used the FX system to demonstrate that real-time visual stimulation can be linked to specific functional states of the brain. They monitored the activity of the brain’s locus coeruleus (LC) norepinephrinergic system in real time. When they visually stimulated the LC neurons during the excitatory phase representing the animal’s awake state, they found that neurons across the brain responded strongly. This result suggests that brain states modulate the processing of visual information, and that closed-loop sensory stimulation could help scientists investigate the interaction between internal brain states and the external environment.

The researchers also used the FX system to demonstrate virtual reality based on an optical brain-machine interface. They reduced the dimensionality of the brain’s neuron activities in real time by establishing multiple neuron ensembles. The closed-loop coupling of these ensembles with the visual environment enabled the researchers to create a virtual reality system that was directly driven by the brain’s neuronal activity.

In this virtual reality system, the gain coupling between the neuronal activity and the environment can be adjusted arbitrarily to allow the neuron ensemble controlling the environment to adjust its output based on gain change.

The researchers will use real-time analysis of big data streams and high-throughput, whole-brain imaging technology to determine the characteristics of neuronal activity that are suitable for optical brain-machine interfaces and will develop more efficient brain-machine interface technologies.

The real-time, big data processing FX system for whole-brain analysis is a step toward improving technologies like virtual reality based on whole-brain, cellular-resolution optical imaging, and optogenetic control. It will also advance closed-loop research in the field of neuroscience.

The research was published in Nature Neuroscience (www.doi.org/10.1038/s41593-024-01595-6).