Generative Language Reconstruction from Brain Recordings

Our study addresses the challenge of reconstructing language from non-invasive brain recordings. Existing methods have addressed this challenge within a classification setup by matching the representation decoded from brain recordings with a limited set of language candidates (typically ranging from 2 to 50). However, these methods cannot guarantee accurate coverage of brain-derived semantics, as the pre-constructed candidate may be inaccurate. To overcome this, we propose a generative method, integrating the representation decoded from functional magnetic resonance imaging (fMRI) directly with a large language model (LLM). Our method can directly generate language outputs from the full vocabulary (32,000 tokens). The language outputs are not only of high quality but also show a remarkable correlation with the auditory or visual stimuli to which brain signals are sampled. Our method achieves a 78.9% win rate compared to its control and exhibits a 25.7% improvement based on the METEOR metric compared to previous approaches within a classification setup. The proposed generative approach can open up a range of potential neurological applications given its ability to decode open-vocabulary language and estimate the generation likelihood of any semantic content. For instance, neurolinguistic research can move beyond designing a limited number of artificially constructed stimuli to employing more natural language stimuli, while also estimating the generation likelihood of any semantic content. The study is accepted by Nature Communications Biology.