Zebrafishtracker3D: A 3D skeleton tracking algorithm for multiple zebrafish based on particle matching

Zebrafish are considered as model organisms in biological and medical research because of their high degree of homology with human genes. Automatic behavioral analysis of multiple zebrafish based on visual tracking is expected to improve research efficiency. However, vision-based multi-object tracking algorithms often suffer from data loss owing to mutual occlusion. In addition, simply tracking zebrafish as points is not sufficient-more detailed information, which is required for research on zebrafish behavior. In this paper, we propose Zebrafishtracker3D, which utilizes a skeleton stability strategy to reduce detection error caused by frequent overlapping of multiple zebrafish effectively and estimates zebrafish skeletons using head coordinates in the top view. Further, we transform the front- and top-view matching task into an optimization problem and propose a particle-matching method to perform 3D tracking. The robustness of the algorithm with respect to occlusion is estimated on the dataset comprising two and three zebrafish. Experimental results demonstrate that the proposed algorithm exhibits a multiple object tracking accuracy (MOTA) exceeding 90% in the top view and a 3D tracking matching accuracy exceeding 90% in the complex videos with frequent overlapping. It is noteworthy that each instance in the trace saves its skeleton. In addition, Zebrafishtracker3D is applied in the zebrafish courtship experiment, establishes the stability of the method in applications of life science, and proves that the data can be used for behavioral analysis. Zebrafishtracker3D is the first algorithm that realizes 3D skeleton tracking of multiple zebrafish simultaneously.