Towards a Real-Time Decoding of Images from Brain Activity

At every moment of every day, our brains meticulously sculpt a wealth of sensory signals into meaningful representations of the world around us. Yet how this continuous process actually works remains poorly understood.

Today, Meta is announcing an important milestone in the pursuit of that fundamental question. Using magnetoencephalography (MEG), a non-invasive neuroimaging technique in which thousands of brain activity measurements are taken per second, we showcase an AI system capable of decoding the unfolding of visual representations in the brain with an unprecedented temporal resolution.

This AI system can be deployed in real time to reconstruct, from brain activity, the images perceived and processed by the brain at each instant. This opens up an important avenue to help the scientific community understand how images are represented in the brain, and then used as foundations of human intelligence. Longer term, it may also provide a stepping stone toward non-invasive brain-computer interfaces in a clinical setting that could help people who, after suffering a brain lesion, have lost their ability to speak.

Leveraging our recent architecture trained to decode speech perception from MEG signals, we develop a three-part system consisting of an image encoder, a brain encoder, and an image decoder. The image encoder builds a rich set of representations of the image independently of the brain. The brain encoder then learns to align MEG signals to these image embeddings. Finally, the image decoder generates a plausible image given these brain representations.

MEG recordings are continuously aligned to the deep representation of the images, which can then condition the generation of images at each instant.

We train this architecture on a public dataset of MEG recordings acquired from healthy volunteers and released by Things, an international consortium of academic researchers sharing experimental data based on the same image database.

We first compare the decoding performance obtained with a variety of pretrained image modules and show that the brain signals best align with modern computer vision AI systems like DINOv2, a recent self-supervised architecture able to learn rich visual representations without any human annotations. This result confirms that self-supervised learning leads AI systems to learn brain-like representations: The artificial neurons in the algorithm tend to be activated similarly to the physical neurons of the brain in response to the same image.

The images that volunteer participants see (left) and those decoded from MEG activity at each instant of time (right). Each image is presented approximately every 1.5 seconds.

This functional alignment between such AI systems and the brain can then be used to guide the generation of an image similar to what the participants see in the scanner. While our results show that images are better decoded with functional Magnetic Resonance Imaging (fMRI), our MEG decoder can be used at every instant of time and thus produces a continuous flux of images decoded from brain activity.

The images that volunteer participants see (left) and those decoded from fMRI activity (right).

While the generated images remain imperfect, the results suggest that the reconstructed image preserves a rich set of high-level features, such as object categories. However, the AI system often generates inaccurate low-level features by misplacing or mis-orienting some objects in the generated images. In particular, using the Natural Scene Dataset, we show that images generated from MEG decoding remain less precise than the decoding obtained with fMRI, a comparably slow-paced but spatially precise neuroimaging technique.

Overall, our results show that MEG can be used to decipher, with millisecond precision, the rise of complex representations generated in the brain. More generally, this research strengthens Meta’s long-term research initiative to understand the foundations of human intelligence, identify its similarities as well as differences compared to current machine learning algorithms, and ultimately guide the development of AI systems designed to learn and reason like humans.

#AIUpskilling #HandsOnAI #AIForAll #LearnAI #AIEducation #AIWorkshops #AITraining #MachineLearning #DeepLearning #DataScience #AICommunity #TechEducation #AIInnovation #TechSkills #FutureOfWork #AIExperts #AIApplications #AIProjects #CareerInAI #TechLeadership #AIEntrepreneur #AIConsulting #AIWorkforce #AIAdvancement #AIForBusiness #AIIndustry #DigitalTransformation #TechTutorials #AIKnowledge #AIProgress

Follow-Us ON

LinkedIn:Follow Let’s Code AI on LinkedIn

Instagram:Follow Let’s Code AI on Instagram

Facebook:Follow Let’s Code AI on Facebook

MediumFollow Lets Code AI on Medium

Recent Post

FAQ's

- The brain decoding process involves analyzing patterns of brain activity to infer or decode specific information, such as thoughts, perceptions, or intentions. Advanced techniques, such as functional magnetic resonance imaging or electroencephalography, are used to record brain activity and decode neural signals.

Brain decoding of images is still under development, but here's the general idea:
- Brain imaging: Techniques like fMRI capture brain activity patterns while a person sees images.
- Machine learning: These algorithms analyze vast amounts of brain activity data paired with the corresponding images.
- Image reconstruction: The trained algorithms learn the relationship between brain activity and image features. They then use this knowledge to reconstruct an image based on new brain activity patterns.

- The visual cortex, located in the occipital lobe at the back of the brain, is primarily responsible for processing and decoding visual information. Different regions within the visual cortex are specialized for processing specific aspects of visual stimuli, such as shape, color, motion, and depth.

This technology holds promise for various applications, including:
- Assistive Technologies: Imagine helping people with sight loss "see" again by decoding visual information from their brain activity.
- Brain-Computer Interfaces: Brain decoding could be used to control external devices or even create virtual reality experiences directly through brain activity.
- Medical Diagnosis: Decoding brain activity could potentially aid in diagnosing neurological disorders that affect vision.

Yes, here are some challenges to consider:
- Accuracy: As mentioned earlier, accuracy needs further improvement for real-world applications.
- Ethical Concerns: Brain decoding raises ethical questions about privacy and potential misuse of brain data.
- Technical Limitations: Brain imaging technologies have limitations in resolution and real-time processing capabilities.

- Decoding complex visual information from the brain poses several challenges, such as decoding fine-grained details, distinguishing between similar objects or categories, accounting for individual differences in brain anatomy and function, and ensuring robustness to noise and artifacts in neural recordings.

- Real-time brain decoding holds promise for clinical applications, such as diagnosing and monitoring neurological disorders, assessing cognitive function, guiding neurorehabilitation therapies, and enhancing communication and control for individuals with severe disabilities.

Brain decoding raises concerns about:
- Privacy: Ensuring the security and privacy of brain data is crucial.
- Informed Consent: People should be fully informed about the use of their brain data for research or clinical applications.
- Potential for Misuse: Measures need to be in place to prevent the misuse of brain decoding technology.

- Improved machine learning algorithms: Advances in AI and machine learning are expected to enhance the accuracy and efficiency of brain decoding.
- Combination of brain imaging techniques: Using multiple brain imaging methods might provide a more comprehensive picture of brain activity, leading to better image reconstructions.
- Non-invasive brain imaging: Developing less invasive brain imaging techniques could make brain decoding more accessible and applicable.

Brain decoding research is rapidly advancing. Future advancements in:
- Machine Learning Techniques: More sophisticated algorithms could lead to higher accuracy in image reconstruction.
- Brain Imaging Technology: Improved resolution and real-time processing capabilities could open doors for new applications.

Scroll to Top
Register For A Course