replicAnt: a pipeline for generating annotated images of animals in complex environments using Unreal Engine

Area of Science:

• Computer Vision
• Animal Behavior
• Machine Learning

Background:

• Deep learning computer vision methods are advancing animal behavior research.
• Transfer learning aids non-model species but requires extensive manual annotation and performs best in controlled settings.

Purpose of the Study:

• To develop a configurable pipeline for generating large, variable training datasets for animal behavior analysis.
• To overcome limitations of manual annotation and improve the performance and robustness of computer vision models in this field.

Main Methods:

• Developed replicAnt, a pipeline using Unreal Engine 5 and Python to create synthetic data.
• Placed 3D animal models in procedurally generated environments to automatically annotate images.
• Utilized consumer-grade hardware for data generation.

Main Results:

• Synthetic data from replicAnt significantly reduced hand-annotation needs for benchmark performance.
• Achieved improvements in animal detection, tracking, pose-estimation, and semantic segmentation.
• Demonstrated increased subject-specificity and domain-invariance, enhancing model robustness.

Conclusions:

• ReplicAnt effectively generates synthetic data, reducing manual annotation for computer vision in animal behavior.
• The pipeline enhances model performance, robustness, and may eliminate the need for manual annotation in some cases.
• Represents a significant advancement for applying deep learning computer vision tools in field research.