NEURAL-VOX: NEURal auditory language decoding for voice and text reconstruction

Area of Science:

• Neuroscience
• Artificial Intelligence
• Assistive Technology

Background:

• Neural decoding of perceived linguistic content from non-invasive brain recordings is a significant challenge.
• Existing methods struggle with intermediate representations (mel spectrograms, phonemes) and multi-modal integration.

Purpose of the Study:

• To present NEURAL-VOX, a framework for decoding non-invasive brain activity into text, phoneme sequences, and mel-spectrogram-based acoustic representations.
• To improve brain-to-text decoding accuracy and enable joint optimization with speech synthesis.

Main Methods:

• Developed NEURAL-VOX with a three-stage training strategy.
• Incorporated multi-scale frequency-domain analysis to capture hierarchical language processing.
• Leveraged multi-modal information for enhanced text decoding.

Main Results:

• NEURAL-VOX achieved substantial gains over existing methods across multiple datasets.
• Learned phoneme representations encode rich linguistic information, strengthening text decoding.
• Model interpretability analysis showed strong alignment with neurobiological patterns.

Conclusions:

• NEURAL-VOX offers a powerful framework for neural decoding of linguistic content.
• The approach enhances assistive technologies by improving accuracy and integrating speech synthesis.
• The model demonstrates effective capture of language processing hierarchies and neurobiological patterns.