Lightning Pose 3D: an uncertainty-aware framework for data-efficient multi-view animal pose estimation

Area of Science:

• Animal behavior quantification
• Biomedical research
• Computer vision

Background:

• Accurate multi-view pose estimation is crucial for analyzing animal behavior in research.
• Current methods face challenges with limited labeled data and unreliable uncertainty estimates.
• Existing techniques often require precise camera calibration, limiting their applicability.

Purpose of the Study:

• To develop a flexible framework for robust multi-view pose estimation.
• To improve tracking accuracy and uncertainty estimation with limited labeled data.
• To enable pose estimation with or without camera calibration.

Main Methods:

• A novel framework combining training and post-processing techniques with uncertainty-aware pseudo-labeling distillation.
• Joint processing of multi-view data using a pretrained vision transformer backbone.
• Simulated occlusion for robust cross-view correspondence learning and optional 3D data augmentation with triangulation-based loss.
• Extension of the Ensemble Kalman Smoother (EKS) for nonlinear cases and a variance inflation technique for inconsistency detection.

Main Results:

• The proposed pipeline consistently outperforms existing methods across five diverse datasets (fly, mouse, bird), including multi-animal scenarios.
• Demonstrated significant improvements in downstream scientific analyses, such as unsupervised behavioral clustering and neural decoding.
• Achieved better performance with as few as 200 labeled frames, highlighting efficiency with limited data.

Conclusions:

• The developed framework offers a significant advancement in multi-view pose estimation for animal behavior research.
• The method's flexibility, accuracy, and improved uncertainty estimation facilitate more reliable scientific discovery.
• A user-friendly interface supports the entire pose estimation workflow, promoting wider adoption and application.