Unsupervised transfer learning enables multi-animal tracking without training annotation

Quantitative ethology necessitates accurate tracking of animal locomotion, especially for population-level analyses involving multiple individuals. However, current methods mostly rely on laborious annotations for supervised training and have restricted performance under challenging conditions. Here we present an unsupervised deep transfer learning method for multi-animal tracking (UDMT) that achieves state-of-the-art performance without requiring training annotations. By synergizing a bidirectional closed-loop tracking strategy, a spatiotemporal transformer network and three dedicated modules for localization refining, bidirectional identity correction and automatic parameter tuning, UDMT can track multiple animals accurately under various challenging conditions, such as crowding, occlusion, rapid motion, low image contrast and cross-species experiments. We demonstrate the versatility of UDMT on five different kinds of model animals, including mice, rats, Drosophila, Caenorhabditis elegans and Betta splendens. Combined with a head-mounted miniaturized microscope, we illustrate the power of UDMT for neuroethological interrogations to decipher the correlations between animal locomotion and neural activity.