Learning neural decoders without labels using multiple data streams.

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Brain-computer interfaces (BCIs) are advancing rapidly for tasks like speech and movement assistance.
• Current BCI training often requires extensive labeled data, limiting real-world application.
• Self-supervised models show promise but their efficacy in neural decoding is uncertain.

Purpose of the Study:

• To develop and evaluate a self-supervised method for training neural decoders using unlabeled, multi-modal data.
• To assess the performance of cross-modal self-supervised learning against supervised and unimodal self-supervised approaches.
• To extend the cross-modal framework to incorporate more than two data streams for enhanced decoding.

Main Methods:

• Applied cross-modal, self-supervised deep clustering to simultaneously recorded neural, kinematic, and physiological signals.
• Trained decoders to classify movements from brain recordings without relying on explicit labels.
• Compared the accuracy of decoders trained via cross-modal self-supervised learning against traditional supervised and unimodal self-supervised methods.

Main Results:

• Cross-modal self-supervised training significantly boosted decoding performance compared to unimodal methods.
• Decoding accuracies achieved through this method closely matched those of supervised decoders.
• Extending the approach to three or more modalities yielded state-of-the-art neural decoding accuracy, matching or exceeding supervised models.

Conclusions:

• Cross-modal, self-supervised decoding is effective for training neural decoders with limited or no labeled data.
• The developed framework can be extended to multiple data streams, enhancing self-supervised training efficacy.
• This research advances BCIs by enabling robust decoder training in data-scarce environments.