Paper List
-
SpikGPT: A High-Accuracy and Interpretable Spiking Attention Framework for Single-Cell Annotation
This paper addresses the core challenge of robust single-cell annotation across heterogeneous datasets with batch effects and the critical need to ide...
-
Unlocking hidden biomolecular conformational landscapes in diffusion models at inference time
This paper addresses the core challenge of efficiently and accurately sampling the conformational landscape of biomolecules from diffusion-based struc...
-
Personalized optimization of pediatric HD-tDCS for dose consistency and target engagement
This paper addresses the critical limitation of one-size-fits-all HD-tDCS protocols in pediatric populations by developing a personalized optimization...
-
Realistic Transition Paths for Large Biomolecular Systems: A Langevin Bridge Approach
This paper addresses the core challenge of generating physically realistic and computationally efficient transition paths between distinct protein con...
-
Consistent Synthetic Sequences Unlock Structural Diversity in Fully Atomistic De Novo Protein Design
This paper addresses the core pain point of low sequence-structure alignment in existing synthetic datasets (e.g., AFDB), which severely limits the pe...
-
MoRSAIK: Sequence Motif Reactor Simulation, Analysis and Inference Kit in Python
This work addresses the computational bottleneck in simulating prebiotic RNA reactor dynamics by developing a Python package that tracks sequence moti...
-
On the Approximation of Phylogenetic Distance Functions by Artificial Neural Networks
This paper addresses the core challenge of developing computationally efficient and scalable neural network architectures that can learn accurate phyl...
-
EcoCast: A Spatio-Temporal Model for Continual Biodiversity and Climate Risk Forecasting
This paper addresses the critical bottleneck in conservation: the lack of timely, high-resolution, near-term forecasts of species distribution shifts ...
Towards unified brain-to-text decoding across speech production and perception
Zhejiang University | Chinese Academy of Sciences | Huashan Hospital, Fudan University
30秒速读
IN SHORT: This paper addresses the core challenge of developing a unified brain-to-text decoding framework that works across both speech production and perception modalities for Mandarin Chinese, overcoming limitations of single-modality approaches and alphabetic language systems.
核心创新
- Methodology First unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese, enabling direct comparison of neural dynamics across modalities.
- Methodology Three-stage post-training and two-stage inference framework for 7B-parameter LLM that outperforms larger commercial LLMs (hundreds of billions of parameters) in mapping toneless Pinyin syllables to Chinese sentences.
- Biology Revealed neural characteristics of Mandarin speech: production engages broader cortical regions than perception; shared channels show similar patterns with perception delayed by ~106.5ms; comparable decoding performance across hemispheres.
主要结论
- Achieved best-case Chinese character error rates of 14.71% for spoken sentences and 21.80% for heard sentences across 12 participants with depth electrodes (mean speaking CER = 31.52%, mean listening CER = 37.28%).
- NeuroSketch (2D-CNN) achieved mean initial/final accuracies of 59.54%/50.17% for speaking and 58.92%/48.05% for listening, representing 394.9%/412.0% and 389.7%/406.6% improvements over chance respectively.
- Speech production involved neural responses across broader cortical regions than auditory perception (p<0.05), with perception showing consistent temporal delay relative to production (mean = -106.5ms, 90% CI [-249.4, 23.05]).
摘要: Speech production and perception constitute two fundamental and distinct modes of human communication. Prior brain-to-text decoding studies have largely focused on a single modality and alphabetic languages. Here, we present a unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese. The framework exhibits strong generalization ability, enabling sentence-level decoding when trained only on single-character data and supporting characters and syllables unseen during training. In addition, it allows direct and controlled comparison of neural dynamics across modalities. We collected neural data from 12 participants implanted with depth electrodes and achieved full-sentence decoding across multiple participants, with best-case Chinese character error rates of 14.71% for spoken sentences and 21.80% for heard sentences. Mandarin speech is decoded by first classifying syllable components in Hanyu Pinyin, namely initials and finals, from neural signals, followed by a post-trained large language model (LLM) that maps sequences of toneless Pinyin syllables to Chinese sentences. To enhance LLM decoding, we designed a three-stage post-training and two-stage inference framework based on a 7-billion-parameter LLM, achieving overall performance that exceeds larger commercial LLMs with hundreds of billions of parameters or more. In addition, several characteristics were observed in Mandarin speech production and perception: speech production involved neural responses across broader cortical regions than auditory perception; channels responsive to both modalities exhibited similar activity patterns, with speech perception showing a temporal delay relative to production; and decoding performance was broadly comparable across hemispheres. Our work not only establishes the feasibility of a unified decoding framework but also provides insights into the neural characteristics of Mandarin speech production and perception. These advances contribute to brain-to-text decoding in logosyllabic languages and pave the way toward neural language decoding systems supporting multiple modalities.