Self-Supervised Keypoint Discovery in Behavioral Videos.

Area of Science:

• Computer Vision
• Machine Learning
• Bioinformatics

Background:

• Learning agent posture and structure from unlabeled videos is challenging.
• Existing methods often require manual annotations, increasing costs and time.
• Agent movement is a primary source of information in behavioral videos.

Purpose of the Study:

• To propose a novel self-supervised method for discovering keypoints representing agent posture and structure from unlabeled behavioral videos.
• To demonstrate the generality and effectiveness of the proposed method across diverse agent types.
• To reduce the reliance on manual annotations in training machine learning models for behavioral analysis.

Main Methods:

• Behavioral Keypoint Discovery (B-KinD) utilizes an encoder-decoder architecture with a geometric bottleneck.
• The method reconstructs the spatiotemporal difference between video frames, focusing on regions of movement.
• It operates directly on input videos without requiring manual annotations.

Main Results:

• B-KinD successfully discovers semantically meaningful keypoints across various agents (mouse, fly, human, jellyfish, trees).
• The method achieves state-of-the-art performance in keypoint regression among self-supervised approaches.
• Discovered keypoints perform comparably to supervised keypoints in downstream tasks like behavior classification.

Conclusions:

• B-KinD offers a powerful self-supervised approach for extracting structural and postural information from unlabeled behavioral videos.
• The method significantly reduces the cost and effort associated with model training compared to supervised methods.
• This technique has broad applicability in analyzing animal behavior, biomechanics, and plant movement.