Improved evaluation of waveform reconstruction in speech decoding based on invasive brain-computer interfaces

Brain-computer interfaces (BCIs) that reconstruct speech waveforms from neural signals are a promising communication technology. However, the field lacks a standardized evaluation metric, making it difficult to compare results across studies. Existing objective metrics, such as correlation coefficient (CC) and mel cepstral distortion (MCD), are often used inconsistently and have intrinsic limitations. This study addresses the critical need for a robust and validated method for evaluating reconstructed waveform quality. Literature about waveform reconstruction from intracranial signals is reviewed, and issues with evaluation methods are presented. We collated reconstructed audio from 10 published speech BCI studies and collected Mean Opinion Scores (MOS) from human raters to serve as a perceptual ground truth. We then systematically evaluated how well combinations of existing objective metrics (STOI and MCD) could predict these MOS scores. To ensure robustness and generalizability, we employed a rigorous leave-one-dataset-out cross-validation scheme and compared multiple models, including linear and non-linear regressors. This work, for the first time, identifies a lack of a standard evaluation method, which prohibits cross-study comparison. Using 10 public datasets, our analysis reveals that a non-linear model, specifically a Random Forest regressor, provides the most accurate and reliable prediction of subjective MOS ratings (R² = 0.892). We propose this cross-validated Random Forest model, which maps STOI and MCD to a predicted MOS score, as a standardized objective evaluation metric for the speech BCI field. Its demonstrated accuracy and robust validation outperform the available methods. Moreover, it can provide the community with a reliable tool to benchmark performance, facilitate meaningful cross-study comparisons for the first time, and accelerate progress in speech neuroprosthetics.