A Hybrid YOLO and Segment Anything Model Pipeline for Multi-Damage Segmentation in UAV Inspection Imagery

Area of Science:

• Computer Vision
• Artificial Intelligence
• Civil Engineering

Background:

• Automated inspection of civil infrastructure using Unmanned Aerial Vehicles (UAVs) faces challenges in accurately segmenting multiple damage types in high-resolution images.
• Foundational models like the Segment Anything Model (SAM) show promise for data-efficient segmentation but require effective prompting strategies, especially for complex defects.

Purpose of the Study:

• To conduct a comparative analysis of deep learning strategies for segmenting cracks, efflorescences, and exposed rebars in infrastructure imagery.
• To identify an optimal deep learning pipeline for enhanced automated damage detection and segmentation.

Main Methods:

• Systematic evaluation of three end-to-end segmentation frameworks: YOLOv11 native output, Segment Anything Model (SAM) with bounding box prompts, and SAM with point prompts derived from detector probability maps.
• Development of a hybrid pipeline utilizing the native segmentation of a SAHI-trained YOLO model for cracks and SAM prompted by YOLO bounding boxes for efflorescence and exposed rebar.

Main Results:

• The optimized hybrid pipeline achieved a mean Average Precision (mAP50) of 0.593.
• Class-specific Intersection over Union (IoU) scores were 0.495 for cracks, 0.331 for efflorescence, and 0.205 for exposed rebar.
• A class-specific strategy leveraging the strengths of specialized detectors and foundation models proved most effective.

Conclusions:

• Intelligent frameworks that combine specialized detectors and powerful foundation models in a context-aware manner are crucial for the future of automated infrastructure inspection.
• The proposed hybrid approach demonstrates a significant advancement in accurately segmenting diverse infrastructure damages.