Enhancing YOLO for laparoscopic tool detection: novel data augmentation and structural modifications addressing...

Laparoscopic tool detection is vital for assistance of minimally invasive surgeries, aiding tasks like tool pose estimation and surgical navigation. This study enhances YOLO models for better detection of bifurcated targets (BT) in such procedures, addressing the issue of mis-detection of bifurcated targets (MDBT) where BT tips are misidentified as separate entities or overlooked.

We proposed a data augmentation strategy, Random Target Masking, to prevent the model from identifying BT tips as separate laparoscopic tools. Mixup Plus was developed to balance instance count across categories with varying BT proportions. Additionally, we employed the Space-to-Depth Convolution block for downsampling to curb the model's tendency to overlook small-sized BT tips.

The YOLOv8 model featuring our modifications, tested on our dataset derived from EndoVis17 and EndoVis18, showed improvement in both ${\mathrm{mAP}}_{50}$ and ${\mathrm{mAP}}_{50:95}$ metrics on the test dataset. On the BT test dataset specifically, ${\mathrm{mAP}}_{50}$ and ${\mathrm{mAP}}_{50:95}$ metrics improved by nearly 0.2 and 0.1, respectively. For the Clip Applier category, which has the fewest instances (fewer than 100 instances in the entire training and test dataset), the YOLOv8n model incorporating our proposed modifications increased ${\mathrm{AP}}_{50}$ from 0.0251 to 0.457.

This study focused on improving BT detection accuracy in laparoscopic tool detection using YOLO models, incorporating RTM and MUP data augmentation techniques along with SPD-Conv block integration. Experimental evaluations based on the EndoVis datasets validated the enhancements. The ablation study confirmed the effectiveness of each proposed improvement, particularly highlighting the distinct advantages of the proposed data augmentation methods.