YOLOv8-TF: Transformer-Enhanced YOLOv8 for Underwater Fish Species Recognition with Class Imbalance Handling

In video-based fish surveys, species recognition plays a vital role in stock assessments, ecosystem analysis, production management, and protection of endangered species. However, implementing fish species detection algorithms in underwater environments presents significant challenges due to factors such as varying lighting conditions, water turbidity, and the diverse appearances of fish species. In this work, a transformer-enhanced YOLOv8 (YOLOv8-TF) is proposed for underwater fish species recognition. The YOLOv8-TF enhances the performance of YOLOv8 by adjusting depth scales, incorporating a transformer block into the backbone and neck, and introducing a class-aware loss function to address class imbalance in the dataset. The class-aware loss considers the count of instances within each species and assigns a higher weight to species with fewer instances. This approach enables fish species recognition through object detection, encompassing the classification of each fish species and localization to estimate their position and size within an image. Experiments were conducted using the 2021 Southeast Area Monitoring and Assessment Program (SEAMAPD21) dataset, a detailed and extensive reef fish dataset from the Gulf of Mexico. The experimental results on SEAMAPD21 demonstrate that the YOLOv8-TF model, with a mean Average Precision (mAP)_0.5 of 87.9% and mAP_0.5-0.95 of 61.2%, achieves better detection results for underwater fish species recognition compared to state-of-the-art YOLO models. Additionally, experimental results on the publicly available datasets, such as Pascal VOC and MS COCO datasets demonstrate that the model outperforms existing approaches.