Leveraging Vision Foundation Model via PConv-Based Fine-Tuning with Automated Prompter for Defect Segmentation

Area of Science:

• Computer Vision
• Machine Learning
• Industrial Automation

Background:

• Image segmentation is critical for industrial defect detection.
• Foundation models like Segment Anything Model (SAM) offer strong generalization but face challenges in industrial settings due to feature discrepancies and reliance on manual prompts.
• Existing SAM applications struggle with scalability in specialized industrial environments.

Purpose of the Study:

• To develop an industrial defect segmentation framework, PA-SAM, that overcomes the limitations of SAM for industrial applications.
• To enhance SAM's performance on industrial defect datasets through specialized fine-tuning and automated prompt generation.
• To adapt SAM into an end-to-end semantic segmentation solution for industrial defect identification.

Main Methods:

• Introduced Multi-Scale Partial Convolution Aggregation (MSPCA) within Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning (PEFT) to adapt the image encoder to industrial defect characteristics.
• Developed an Image-to-Prompt Embedding Generator (IPEG) to autonomously create high-quality prompt embeddings from image embeddings, removing the need for manual prompts.
• Refined SAM's mask decoder to create an end-to-end semantic segmentation framework.

Main Results:

• PA-SAM achieved a mean Intersection over Union (IoU) of 73.87% and 68.30% on two real-world industrial defect datasets.
• The framework obtained mean Dice coefficients of 84.90% and 80.22%, outperforming state-of-the-art algorithms.
• Demonstrated robust generalization capabilities and significant application potential in industrial defect segmentation.

Conclusions:

• PA-SAM effectively addresses the limitations of SAM for industrial defect segmentation by integrating PEFT with MSPCA-LoRA and IPEG.
• The proposed framework significantly improves segmentation accuracy and efficiency, offering a scalable solution for industrial defect identification.
• PA-SAM shows strong potential for real-world deployment in industrial quality control and inspection.