Decoding Handwriting Trajectories from Intracortical Brain Signals for Brain-to-Text Communication

The potential to decode handwriting trajectories from brain signals has yet to be fully explored in clinical brain-computer interfaces (BCIs). Here, intracortical neural signals are recorded from a paralyzed individual during attempted handwriting of complex characters. An innovative decoding framework is introduced to address both shape and temporal distortions between neural activity and movement, effectively resolving the misalignment issue commonly encountered in clinical BCIs due to the lack of accurate movement labels. The results demonstrated the reconstruction of highly accurate and human-recognizable handwriting trajectories, significantly outperforming conventional methods. Furthermore, the new framework enabled effective multi-day data fusion, leading to additional improvements in trajectory quality. By employing a dynamic time warping approach to translate trajectories into text, a recognition rate up to 91.1% is achieved within a 1000-character database. Additionally, the framework is applied to reconstruct single-trial trajectories of English letters using a previously published dataset, achieving similarly high recognition rates. Collectively, these findings present a novel BCI decoding scheme capable of accurately reconstructing handwriting trajectories, demonstrating its applicability to both alphabetic and logographic brain-to-text translation. This approach has the potential to revolutionize communication for individuals with motor impairments by enabling accurate brain-to-text translation across diverse languages.