A Novel Fish Pose Estimation Method Based on Semi-Supervised Temporal Context Network

Area of Science:

• Robotics and Autonomous Systems
• Computer Vision
• Marine Biology

Background:

• Biomimetic robotic fish are crucial for ocean exploration in challenging environments.
• Accurate fish pose estimation is vital for analyzing swimming patterns and informing robot design.
• A scarcity of high-quality motion datasets hinders progress in this field.

Purpose of the Study:

• To address the lack of annotated motion data for fish pose estimation.
• To develop an advanced pose estimation method that overcomes limitations of existing approaches.
• To enable more detailed analysis of fish swimming behaviors for ecological studies and robotic development.

Main Methods:

• A custom dual-camera experimental platform was created to capture multi-view carp swimming sequences.
• A novel Semi-supervised Temporal Context-Aware Network (STC-Net) was proposed.
• STC-Net utilizes unsupervised loss functions (temporal continuity, pose plausibility) and a Bi-directional Convolutional Recurrent Neural Network for spatio-temporal modeling.

Main Results:

• The proposed STC-Net achieved a keypoint detection Root Mean Square Error (RMSE) of 9.71 on the custom dataset.
• The network effectively leverages both annotated and unannotated data, improving robustness.
• The method demonstrates computational efficiency and end-to-end trainability.

Conclusions:

• STC-Net offers a robust and scalable solution for biological pose estimation in complex underwater scenarios.
• The developed dataset and method advance the capabilities of underwater robotic fish for exploration and observation.
• This work contributes to improved understanding of fish locomotion and bio-inspired robot design.