Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Signal Processing

Background:

• Deep learning for neurological signals promises advancements in diagnostics, neurorehabilitation, and brain-computer interfaces.
• Acquiring extensive, high-quality annotated neurological data is a significant bottleneck due to cost and expertise requirements.

Purpose of the Study:

• To introduce Neuro-BERT, a self-supervised pre-training framework for neurological signals to address data scarcity in deep learning.
• To leverage frequency and phase information within neurological signals for enhanced representation learning.

Main Methods:

• Developed Neuro-BERT, a self-supervised pre-training framework utilizing masked autoencoding in the Fourier domain.
• Introduced a novel pre-training task, Fourier Inversion Prediction (FIP), to reconstruct masked portions of neurological signals.
• Employed a simple transformer encoder without requiring data augmentation, unlike contrastive methods.

Main Results:

• Neuro-BERT effectively learns from neurological signals by analyzing their frequency and phase distributions.
• The pre-trained models demonstrate significant improvements across various downstream neurological tasks.
• The method shows robust performance without the need for complex data augmentation strategies.

Conclusions:

• Neuro-BERT offers an effective solution for data-scarce scenarios in deep learning for neurological signals.
• The Fourier domain approach and FIP task provide a powerful pre-training strategy.
• This framework has broad applicability in medical diagnostics, neurorehabilitation, and brain-computer interfaces.