Chaofei Fan

All Publications

• Brain-to-text decoding with context-aware neural representations and large language models. Journal of neural engineering Li, J., Le, T., Fan, C., Chen, M., Shlizerman, E. 2025

  Abstract

  Decoding attempted speech from neural activity offers a promising avenue for restoring communication abilities in individuals with speech impairments. Previous studies have focused on mapping neural activity to text using phonemes as the intermediate target. While successful, decoding neural activity directly to phonemes ignores the context dependent nature of the neural activity-to-phoneme mapping in the brain, leading to suboptimal decoding performance. In this work, we propose the use of diphone - an acoustic representation that captures the transitions between two phonemes - as the context-aware modeling target. We integrate diphones into existing phoneme decoding frameworks through a novel divide-and-conquer strategy in which we model the phoneme distribution by marginalizing over the diphone distribution. Our approach effectively leverages the enhanced context-aware representation of diphones while preserving the manageable class size of phonemes, a key factor in simplifying the subsequent phoneme-to-text conversion task. We demonstrate the effectiveness of our approach on the Brain-to-Text 2024 benchmark, where it achieves state-of-the-art Phoneme Error Rate (PER) of 15.34% compared to 16.62% PER of monophone-based decoding. When coupled with finetuned Large Language Models (LLMs), our method yields a Word Error Rate (WER) of 5.77%, significantly outperforming the 8.93% WER of the leading method in the benchmark.

  View details for DOI 10.1088/1741-2552/adfab1

  View details for PubMedID 40795874

• Error encoding in human speech motor cortex. bioRxiv : the preprint server for biology Hou, X., Iacobacci, C., Card, N. S., Wairagkar, M., Singer-Clark, T., Kunz, E. M., Fan, C., Kamdar, F., Hahn, N., Hochberg, L. R., Henderson, J. M., Willett, F. R., Brandman, D. M., Stavisky, S. D. 2025

  Abstract

  Humans monitor their actions, including detecting errors during speech production. This self-monitoring capability also enables speech neuroprosthesis users to recognize mistakes in decoded output upon receiving visual or auditory feedback. However, it remains unknown whether neural activity related to error detection is present in the speech motor cortex. In this study, we demonstrate the existence of neural error signals in speech motor cortex firing rates during intracortical brain-to-text speech neuroprosthesis use. This activity could be decoded to enable the neuroprosthesis to identify its own errors with up to 86% accuracy. Additionally, we observed distinct neural patterns associated with specific types of mistakes, such as phonemic or semantic differences between the person's intended and displayed words. These findings reveal how feedback errors are represented within the speech motor cortex, and suggest strategies for leveraging these additional cognitive signals to improve neuroprostheses.

  View details for DOI 10.1101/2025.06.07.658426

  View details for PubMedID 40661574

  View details for PubMedCentralID PMC12259010

• An Accurate and Rapidly Calibrating Speech Neuroprosthesis. The New England journal of medicine Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., Srinivasan, A., Choi, E. Y., Glasser, M. F., Hochberg, L. R., Henderson, J. M., Shahlaie, K., Stavisky, S. D., Brandman, D. M. 2024; 391 (7): 609-618

  Abstract

  Brain-computer interfaces can enable communication for people with paralysis by transforming cortical activity associated with attempted speech into text on a computer screen. Communication with brain-computer interfaces has been restricted by extensive training requirements and limited accuracy.A 45-year-old man with amyotrophic lateral sclerosis (ALS) with tetraparesis and severe dysarthria underwent surgical implantation of four microelectrode arrays into his left ventral precentral gyrus 5 years after the onset of the illness; these arrays recorded neural activity from 256 intracortical electrodes. We report the results of decoding his cortical neural activity as he attempted to speak in both prompted and unstructured conversational contexts. Decoded words were displayed on a screen and then vocalized with the use of text-to-speech software designed to sound like his pre-ALS voice.On the first day of use (25 days after surgery), the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. Calibration of the neuroprosthesis required 30 minutes of cortical recordings while the participant attempted to speak, followed by subsequent processing. On the second day, after 1.4 additional hours of system training, the neuroprosthesis achieved 90.2% accuracy using a 125,000-word vocabulary. With further training data, the neuroprosthesis sustained 97.5% accuracy over a period of 8.4 months after surgical implantation, and the participant used it to communicate in self-paced conversations at a rate of approximately 32 words per minute for more than 248 cumulative hours.In a person with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore conversational communication after brief training. (Funded by the Office of the Assistant Secretary of Defense for Health Affairs and others; BrainGate2 ClinicalTrials.gov number, NCT00912041.).

  View details for DOI 10.1056/NEJMoa2314132

  View details for PubMedID 39141853

• A flexible intracortical brain-computer interface for typing using finger movements. bioRxiv : the preprint server for biology Shah, N. P., Willsey, M. S., Hahn, N., Kamdar, F., Avansino, D. T., Fan, C., Hochberg, L. R., Willett, F. R., Henderson, J. M. 2024

  Abstract

  Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is "point-and-click" typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is "keystroke" typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.

  View details for DOI 10.1101/2024.04.22.590630

  View details for PubMedID 38712189

• Towards a Quantitative Analysis of Coarticulation with a Phoneme-to-Articulatory Model Fan, C., Henderson, J. M., Manning, C., Willett, F. R., Int Speech Commun Assoc ISCA-INT SPEECH COMMUNICATION ASSOC. 2024: 3095-3099

  View details for DOI 10.21437/Interspeech.2024-2191

  View details for Web of Science ID 001331850103042

• A high-performance speech neuroprosthesis NATURE Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., Shenoy, K. V., Henderson, J. M. 2023

  View details for DOI 10.1038/s41586-023-06377

  View details for Web of Science ID 001074472200013

• A high-performance speech neuroprosthesis. Nature Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., Shenoy, K. V., Henderson, J. M. 2023

  Abstract

  Speech brain-computer interfaces (BCIs) have the potential to restore rapid communication to people with paralysis by decoding neural activity evoked by attempted speech into text1,2 or sound3,4. Early demonstrations, although promising, have not yet achieved accuracies sufficiently high for communication of unconstrained sentences from a large vocabulary1-7. Here we demonstrate a speech-to-text BCI that records spiking activity from intracortical microelectrode arrays. Enabled by these high-resolution recordings, our study participant-who can no longer speak intelligibly owing to amyotrophic lateral sclerosis-achieved a 9.1% word error rate on a 50-word vocabulary (2.7 times fewer errors than the previous state-of-the-art speech BCI2) and a 23.8% word error rate on a 125,000-word vocabulary (the first successful demonstration, to our knowledge, of large-vocabulary decoding). Our participant's attempted speech was decoded  at 62 words per minute, which is 3.4 times as fast as the previous record8 and begins to approach the speed of natural conversation (160 words per minute9). Finally, we highlight two aspects of the neural code for speech that are encouraging for speech BCIs: spatially intermixed tuning to speech articulators that makes accurate decoding possible from only a small region of cortex, and a detailed articulatory representation of phonemes that persists years after paralysis. These results show a feasible path forward for restoring rapid communication to people with paralysis who can no longer speak.

  View details for DOI 10.1038/s41586-023-06377-x

  View details for PubMedID 37612500

  View details for PubMedCentralID 4464168

• Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication Fan, C., Hahn, N., Kamdar, F., Avansino, D., Wilson, G. H., Hochberg, L., Shenoy, K. V., Henderson, J. M., Willett, F. R. edited by Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2023

  View details for Web of Science ID 001227224008014