Beast3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting

Area of Science:

• Computational Biology
• Computer Vision
• Animal Behavior Analysis

Background:

• Multi-view video is crucial for 3D animal movement analysis.
• Current methods like supervised pose estimation and general 3D reconstruction struggle with specialized lab data and sparse views.

Purpose of the Study:

• To develop a self-supervised framework (BEAST3D) for learning 3D visual representations from unlabeled multi-view animal video.
• To overcome limitations of manual annotation and general-purpose 3D models in laboratory settings.

Main Methods:

• BEAST3D utilizes a vision transformer to predict 3D Gaussian splats from calibrated multi-view video.
• Differentiable rendering reconstructs novel views, and simultaneous segmentation isolates the animal.
• The framework conditions on known camera parameters, enabling reconstruction from as few as four views.

Main Results:

• BEAST3D effectively reconstructs 3D structure and generates viewpoint-invariant features across four species.
• Features transferred successfully to novel view synthesis, multi-view pose estimation, and neural encoding tasks.
• Demonstrated high-quality 3D representations for behavioral analysis.

Conclusions:

• BEAST3D provides a versatile, self-supervised framework for 3D behavioral analysis using multi-view laboratory recordings.
• It reduces the need for manual annotation and adapts to sparse-view conditions.
• Enables advanced analysis of animal behavior by leveraging learned 3D structure.