Brain Activity Decoder Can Reveal Stories in People’s Minds

May 1, 2023 • by Marc Airhart

The work relies in part on a transformer model, similar to the ones that power ChatGPT.

Three researchers peer inside an MRI machine

Alex Huth (left), Shailee Jain (center) and Jerry Tang (right) prepare to collect brain activity data in the Biomedical Imaging Center at The University of Texas at Austin. The researchers trained their semantic decoder on dozens of hours of brain activity data from participants, collected in an fMRI scanner. Photo Credit: Nolan Zunk/University of Texas at Austin.


A new artificial intelligence system called a semantic decoder can translate a person’s brain activity — while listening to a story or silently imagining telling a story — into a continuous stream of text. The system developed by researchers at The University of Texas at Austin might help people who are mentally conscious yet unable to physically speak, such as those debilitated by strokes, to communicate intelligibly again.

Comparison of short text passages being listened to by a particpant and text coming out of the decoder

This image shows decoder predictions from brain recordings collected while a user listened to four stories. Example segments were manually selected and annotated to demonstrate typical decoder behaviors. The decoder exactly reproduces some words and phrases and captures the gist of many more. Credit: University of Texas at Austin.

A man places head gear on a participant preparing to go into an MRI scanner

Ph.D. student Jerry Tang prepares to collect brain activity data in the Biomedical Imaging Center at The University of Texas at Austin. The researchers trained their semantic decoder on dozens of hours of brain activity data from participants, collected in an fMRI scanner. Photo Credit: Nolan Zunk/University of Texas at Austin.

Three scientists discussing their work at a computer station

Alex Huth (left), discusses the semantic decoder project with Jerry Tang (center) and Shailee Jain (right) in the Biomedical Imaging Center at The University of Texas at Austin. The researchers trained their semantic decoder on dozens of hours of brain activity data from participants, collected in an fMRI scanner. Photo Credit: Nolan Zunk/University of Texas at Austin.

Share


A scientist sits at a desk writing

Department of Computer Science

University of Texas Theoretical Computer Scientist Wins Gödel Prize

A professor in a UT electrical and computer engineering shirt points at a screen with scientific imagery, as two seated students and another researcher smile and look on.

Cockrell School of Engineering

Can AI Make Critical Communications Chips Easier to Design?

An illustration of a seagrass meadow, with the sediment cross-sectioned to reveal roots. There are two small black microhpones resting on the top of the sediment, connected by a cable running back to a metal box.

Podcast

Can Tiny Bubbles Help Save the Planet?