figshare
Browse

Generative Language Reconstruction from Brain Recordings

Version 3 2025-02-12, 02:35
Version 2 2025-02-11, 07:07
Version 1 2025-02-08, 14:28
figure
posted on 2025-02-12, 02:35 authored by Ziyi YeZiyi Ye, Qingyao Ai, Yiqun Liu, Maarten de Rijke, Min Zhang, Christina LiomaChristina Lioma, Tuukka Ruotsalo

Our study addresses the challenge of reconstructing language from non-invasive brain recordings. Existing methods have addressed this challenge within a classification setup by matching the representation decoded from brain recordings with a limited set of language candidates (typically ranging from 2 to 50). However, these methods cannot guarantee accurate coverage of brain-derived semantics, as the pre-constructed candidate may be inaccurate. To overcome this, we propose a generative method, integrating the representation decoded from functional magnetic resonance imaging (fMRI) directly with a large language model (LLM). Our method can directly generate language outputs from the full vocabulary (32,000 tokens). The language outputs are not only of high quality but also show a remarkable correlation with the auditory or visual stimuli to which brain signals are sampled. Our method achieves a 78.9% win rate compared to its control and exhibits a 25.7% improvement based on the METEOR metric compared to previous approaches within a classification setup. The proposed generative approach can open up a range of potential neurological applications given its ability to decode open-vocabulary language and estimate the generation likelihood of any semantic content. For instance, neurolinguistic research can move beyond designing a limited number of artificially constructed stimuli to employing more natural language stimuli, while also estimating the generation likelihood of any semantic content. The study is accepted by Nature Communications Biology.

Funding

Quan Cheng Laboratory (Grant No. QCLZD202301)

the Horizon 2020 FET program of the EU through the ERA-NET Cofund funding grant CHIST-ERA-20-BCI-001

History

Usage metrics

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC