Guiding Brain-to-Vocalization Decoder Design Using Structured Generalization Error.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Machine Learning

Background:

• Current neuroprostheses decode speech from neural activity, achieving high rates for limited vocabularies.
• Generalizing these decoders to naturalistic speech with large vocabularies and diverse contexts remains a challenge.
• Understanding neural dynamics is key to improving decoder performance.

Purpose of the Study:

• To evaluate neural decoder generalization for naturalistic speech using a novel vocal-unit-level test.
• To compare the performance of decoders using spike trains versus latent neural features (factors and firing rates).
• To investigate the role of Latent Factor Analysis via Dynamical Systems (LFADS) in capturing neural dynamics for vocalization.

Main Methods:

• Developed a vocal-unit-level generalization test for neural decoders.
• Modeled zebra finch vocalization as an analog for human speech production.
• Compared three decoder types: spike trains, neural factors, and firing rates, with factors/rates inferred by LFADS.
• Assessed generalization using both random holdout and vocal-unit-holdout error measures.

Main Results:

• Conventional generalization error was similar across all decoder types.
• Decoders using latent neural factors and firing rates significantly outperformed spike-based decoders on vocal-unit-holdout generalization.
• LFADS-derived features better capture flexible vocalization inference from partial data variation.

Conclusions:

• Latent neural dynamics, inferred by LFADS, are crucial for robust speech decoding generalization.
• Factor- and rate-based decoders offer improved adaptability for naturalistic speech applications.
• Future neuroprosthetic development should incorporate latent neural and vocalization dynamics for enhanced communication.