Paper List
-
GOPHER: Optimization-based Phenotype Randomization for Genome-Wide Association Studies with Differential Privacy
This paper addresses the core challenge of balancing rigorous privacy protection with data utility when releasing full GWAS summary statistics, overco...
-
Real-time Cricket Sorting By Sex A low-cost embedded solution using YOLOv8 and Raspberry Pi
This paper addresses the critical bottleneck in industrial insect farming: the lack of automated, real-time sex sorting systems for Acheta domesticus ...
-
Training Dynamics of Learning 3D-Rotational Equivariance
This work addresses the core dilemma of whether to use computationally expensive equivariant architectures or faster symmetry-agnostic models with dat...
-
Fast and Accurate Node-Age Estimation Under Fossil Calibration Uncertainty Using the Adjusted Pairwise Likelihood
This paper addresses the dual challenge of computational inefficiency and sensitivity to fossil calibration errors in Bayesian divergence time estimat...
-
Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training
This paper addresses the core challenge of accurately predicting protein fitness with only a handful of experimental observations, where data collecti...
-
scCluBench: Comprehensive Benchmarking of Clustering Algorithms for Single-Cell RNA Sequencing
This paper addresses the critical gap of fragmented and non-standardized benchmarking in single-cell RNA-seq clustering, which hinders objective compa...
-
Simulation and inference methods for non-Markovian stochastic biochemical reaction networks
This paper addresses the computational bottleneck of simulating and performing Bayesian inference for non-Markovian biochemical systems with history-d...
-
Assessment of Simulation-based Inference Methods for Stochastic Compartmental Models
This paper addresses the core challenge of performing accurate Bayesian parameter inference for stochastic epidemic models when the likelihood functio...
Towards unified brain-to-text decoding across speech production and perception
Zhejiang University | Chinese Academy of Sciences | Huashan Hospital, Fudan University
30秒速读
IN SHORT: This paper addresses the core challenge of developing a unified brain-to-text decoding framework that works across both speech production and perception modalities for Mandarin Chinese, overcoming limitations of single-modality approaches and alphabetic language systems.
核心创新
- Methodology First unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese, enabling direct comparison of neural dynamics across modalities.
- Methodology Three-stage post-training and two-stage inference framework for 7B-parameter LLM that outperforms larger commercial LLMs (hundreds of billions of parameters) in mapping toneless Pinyin syllables to Chinese sentences.
- Biology Revealed neural characteristics of Mandarin speech: production engages broader cortical regions than perception; shared channels show similar patterns with perception delayed by ~106.5ms; comparable decoding performance across hemispheres.
主要结论
- Achieved best-case Chinese character error rates of 14.71% for spoken sentences and 21.80% for heard sentences across 12 participants with depth electrodes (mean speaking CER = 31.52%, mean listening CER = 37.28%).
- NeuroSketch (2D-CNN) achieved mean initial/final accuracies of 59.54%/50.17% for speaking and 58.92%/48.05% for listening, representing 394.9%/412.0% and 389.7%/406.6% improvements over chance respectively.
- Speech production involved neural responses across broader cortical regions than auditory perception (p<0.05), with perception showing consistent temporal delay relative to production (mean = -106.5ms, 90% CI [-249.4, 23.05]).
摘要: Speech production and perception constitute two fundamental and distinct modes of human communication. Prior brain-to-text decoding studies have largely focused on a single modality and alphabetic languages. Here, we present a unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese. The framework exhibits strong generalization ability, enabling sentence-level decoding when trained only on single-character data and supporting characters and syllables unseen during training. In addition, it allows direct and controlled comparison of neural dynamics across modalities. We collected neural data from 12 participants implanted with depth electrodes and achieved full-sentence decoding across multiple participants, with best-case Chinese character error rates of 14.71% for spoken sentences and 21.80% for heard sentences. Mandarin speech is decoded by first classifying syllable components in Hanyu Pinyin, namely initials and finals, from neural signals, followed by a post-trained large language model (LLM) that maps sequences of toneless Pinyin syllables to Chinese sentences. To enhance LLM decoding, we designed a three-stage post-training and two-stage inference framework based on a 7-billion-parameter LLM, achieving overall performance that exceeds larger commercial LLMs with hundreds of billions of parameters or more. In addition, several characteristics were observed in Mandarin speech production and perception: speech production involved neural responses across broader cortical regions than auditory perception; channels responsive to both modalities exhibited similar activity patterns, with speech perception showing a temporal delay relative to production; and decoding performance was broadly comparable across hemispheres. Our work not only establishes the feasibility of a unified decoding framework but also provides insights into the neural characteristics of Mandarin speech production and perception. These advances contribute to brain-to-text decoding in logosyllabic languages and pave the way toward neural language decoding systems supporting multiple modalities.