Identity-stable multi-animal tracking using bidirectional segmentation with object-level memory.

Area of Science:

• Ethology
• Computational Biology
• Neuroscience

Background:

• Accurate tracking of animal behavior is crucial for understanding social dynamics and neural mechanisms.
• Current pose estimation methods face challenges in maintaining consistent identity tracking, especially during complex social interactions.
• Manual correction of identity errors in long-term animal tracking is prohibitively costly and time-consuming.

Purpose of the Study:

• To develop and validate a novel pipeline for bidirectional video object segmentation (VOS) to correct identity swaps in animal tracking.
• To reduce the manual annotation effort required for identity correction in pose estimation data.
• To enable scalable, long-term, and identity-error-free tracking of unmarked animals in group settings.

Main Methods:

• Utilized a state-of-the-art VOS algorithm (Cutie) with multi-timescale memory.
• Implemented bidirectional inference (forward and reverse) to compare segmentation masks.
• Identified and flagged localized zones of disagreement for efficient manual review.

Main Results:

• Reduced identity swaps by two orders of magnitude compared to standard pose estimation workflows.
• Achieved identity error-free segmentation masks and aligned keypoints with review of <0.3% of frames.
• Demonstrated scalability to social interactions involving three or more animals.

Conclusions:

• The developed VOS pipeline offers a practical and scalable solution for long-term tracking of unmarked animals in naturalistic group settings.
• This method significantly lowers the barrier for researchers studying social behavior, enabling more comprehensive and accurate analyses.
• An accessible graphical user interface is provided to facilitate general adoption by the research community.