A spatiotemporal transformer with cross-frame encoding and trajectory-aware decoding for multi-target fish tracking.

Area of Science:

• Computer Vision
• Marine Biology
• Robotics

Background:

• Multi-object tracking in underwater environments is hindered by poor illumination, occlusion, and similar fish appearances.
• Existing methods struggle with the dynamic and complex nature of underwater scenes, impacting tracking accuracy.

Purpose of the Study:

• To develop a unified Transformer framework for accurate and robust multi-object fish tracking in challenging underwater conditions.
• To improve both detection and identity association of fish despite environmental and visual challenges.

Main Methods:

• A Transformer framework integrating cross-frame spatiotemporal encoding and trajectory-aware decoding.
• Incorporation of temporal difference, frame position embeddings, and residual motion enhancement for explicit feature alignment.
• Utilizing trajectory extrapolation priors and temporal association attention for restricted feature aggregation and joint detection-association optimization.

Main Results:

• Achieved MOTA of 0.719, IDF1 of 0.693, and Recall of 0.742 on a self-constructed dataset, outperforming the GTR baseline.
• Attained MOTA of 0.697, IDF1 of 0.680, and Recall of 0.730 on the UOT32 dataset, surpassing ByteTrack.
• Demonstrated significant improvements in tracking accuracy and stability, especially under occlusion and dense target conditions.

Conclusions:

• The proposed framework effectively integrates spatiotemporal modeling and trajectory-guided decoding for superior fish tracking.
• The method shows strong robustness and generalization capabilities in complex underwater environments.
• This approach advances the state-of-the-art in underwater multi-object fish tracking.