EEG-Translator: A Cross-Modality Framework for Subject-Specific EEG and Voice Reconstruction from Imagined Speech

Non-invasive brain-computer interfaces (BCIs) offer the potential to enable communication for individuals with speech impairments by decoding neural signals through speech-related electroencephalography (EEG) signals. Beyond domain-specific speech EEG decoding, generative approaches that enable cross-domain reconstruction are needed to enhance the overall system performance. Here, we propose a cross-modal EEG translation framework that reconstructs overt speech EEG from imagined speech EEG, for subject-specific speech synthesis. Our approach integrates a diffusion-based model with GAN training to enhance cross-domain EEG reconstruction by preserving both EEG class information and its time-frequency domain properties. In classification tasks, the reconstructed EEG improves class decoding accuracy by 6.2% over the original imagined EEG. Additionally, EEG reconstruction was trained not only on the EEG signal itself but also by incorporating spectrogram-based features, leveraging a fusion of spatial and spectral losses to preserve EEG properties. Beyond EEG reconstruction, category-wise analysis across a multi-speech paradigm dataset reveals variations in decoding performance, offering linguistic insights crucial for the advancement of speech BCI systems. Our findings highlight the potential of diffusion-driven EEG translation in speech BCIs, emphasizing the importance of integrating deep learning methodologies with linguistic insights for improved neural signal reconstruction and interpretation.