Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining.

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Understanding the brain requires studying behavior, but current video analysis methods need extensive labeled data.
• This limits behavioral analysis in neuroscience research, especially when labeled datasets are scarce.

Purpose of the Study:

• To introduce BEAST (Behavioral Analysis via Self-supervised pretraining of Transformers), a scalable framework for analyzing neuro-behavioral data.
• To leverage unlabeled video data for pretraining vision transformers for diverse behavioral analyses.

Main Methods:

• BEAST utilizes masked autoencoding and temporal contrastive learning on unlabeled video data.
• It pretrains experiment-specific vision transformers for behavioral analysis tasks.

Main Results:

• BEAST demonstrated improved performance across multiple species in three key tasks: behavioral feature extraction, pose estimation, and action segmentation.
• The framework showed effectiveness in both single- and multi-animal settings.
• BEAST accelerates behavioral analysis in data-scarce scenarios.

Conclusions:

• BEAST provides a powerful and versatile backbone model for behavioral analysis in neuroscience.
• The framework effectively addresses the challenge of limited labeled data in video-based behavioral studies.
• This approach enhances the ability to correlate neural activity with specific behaviors.