A unique blend of brain imaging and computer simulation demonstrated at the University of California has shown the potential for literally recording and displaying one’s thoughts and dreams. Using functional magnetic resonance imaging (fMRI) and computational models, the researchers decoded and reconstructed people’s dynamic visual experiences — in this case, watching Hollywood movie trailers. As yet, the technology can reconstruct only movie clips people have already viewed. However, it may pave the way for reproducing the images inside our heads that no one else sees, such as dreams and memories, according to the researchers. The approximate reconstruction (right) of a movie clip (left) is achieved through brain imaging and computer simulation. (Images: UC Berkeley) “This is a major leap toward reconstructing internal imagery,” said Jack Gallant, co-author of a study published online Sept. 22 in Current Biology. “We are opening a window into the movies in our minds.” Eventually, practical applications of the technology could include a better understanding of what goes on in the minds of people who cannot communicate verbally, such as stroke victims, coma patients and people with neurodegenerative diseases. It also may lay the groundwork for brain-machine interface, so that people with cerebral palsy or paralysis, for example, could guide computers with their minds. However, researchers point out that the technology is decades from allowing users to read others’ thoughts and intentions, as portrayed in a variety of science-fiction films and novels. Previously, Gallant and his colleagues recorded brain activity in the visual cortex while a subject viewed black-and-white photographs. They then built a computational model that enabled them to predict with high accuracy which picture the subject was looking at. In their latest experiment, the investigators say they have solved a much more difficult problem by actually decoding brain signals generated by moving pictures. “Our natural visual experience is like watching a movie,” said Shinji Nishimoto, lead author of the study and a postdoctoral researcher in Gallant’s lab. “For this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences.” Nishimoto and two other team members served as subjects for the experiment because the procedure requires volunteers to remain still inside the MRI scanner for hours at a time. They watched two separate sets of Hollywood movie trailers, while fMRI was used to measure blood flow through the visual cortex, the part of the brain that processes visual information. On the computer, the brain was divided into small, 3-D cubes known as volumetric pixels, or “voxels.” “We built a model for each voxel that describes how shape and motion information in the movie is mapped into brain activity,” Nishimoto said. The brain activity recorded while subjects viewed the first set of clips fed into a computer program that learned, second by second, to associate visual patterns in the movie with the corresponding brain activity. Brain activity evoked by the second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random YouTube videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject. Finally, the 100 clips that the computer program decided were most similar to the clip that the subject had probably seen were merged to produce a blurry yet continuous reconstruction of the original movie. Reconstructing movies using brain scans has been challenging because the blood flow signals measured using fMRI change much more slowly than the neural signals that encode dynamic information in movies, researchers say. For this reason, most previous attempts to decode brain activity have focused on static images. “We addressed this problem by developing a two-stage model that separately describes the underlying neural population and blood flow signals,” Nishimoto said. Ultimately, scientists need to understand how the brain processes dynamic visual events that we experience in everyday life, he added. “We need to know how the brain works in naturalistic conditions,” he said. “For that, we need to first understand how the brain works while we are watching movies.” For more information, visit: www.berkeley.edu