Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native...

Area of Science:

• Ethology and behavioral neuroscience
• Computer vision and machine learning

Background:

• Supervised deep learning for pose estimation requires extensive manual labeling.
• Existing methods can produce unreliable outputs for scientific analysis.

Purpose of the Study:

• Introduce 'Lightning Pose', an efficient package for semi-supervised pose estimation.
• Improve the accuracy and scientific usability of pose estimation trajectories.

Main Methods:

• Utilize semi-supervised learning with labeled and unlabeled video frames.
• Incorporate motion continuity, multi-view geometry, and posture plausibility penalties.
• Employ a novel network architecture for occlusion resolution using surrounding frames.
• Refine predictions using ensembling and Kalman smoothing.

Main Results:

• Achieved more accurate and scientifically usable pose trajectories.
• Reduced the need for extensive manual labeling through semi-supervised learning.

Conclusions:

• Lightning Pose offers an efficient and robust solution for behavioral analysis.
• The developed cloud application facilitates data labeling, network training, and video processing.