LLaFS++: Few-Shot Image Segmentation With Large Language Models

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Few-shot segmentation (FSS) methods struggle with limited labeled data, hindering performance.
• Existing FSS approaches rely on small, potentially biased datasets, restricting generalization capabilities.

Purpose of the Study:

• To introduce LLaFS++, a pioneering framework applying large language models (LLMs) to few-shot segmentation (FSS).
• To overcome the limitations of insufficient and biased information in few-shot labeled samples by leveraging LLM prior knowledge.

Main Methods:

• LLaFS++ integrates LLMs to guide the FSS process, compensating for limited sample information.
• Introduces task-specific designs: polygon output instruction, region-attribute table for multi-modal guidance, pseudo-sample synthesis, curriculum learning, and a novel inference method to prevent oversegmentation.
• Leverages LLM's extensive prior knowledge for superior segmentation guidance.

Main Results:

• LLaFS++ achieves state-of-the-art results on benchmark datasets: PASCAL-$5^{i}$5i, COCO-$20^{i}$20i, and FSS-1000.
• Demonstrates significant performance improvements by effectively utilizing LLM guidance.
• The framework successfully mitigates oversegmentation hallucinations through its novel inference method.

Conclusions:

• LLaFS++ represents a significant advancement in few-shot segmentation by successfully integrating large language models.
• The proposed framework showcases the remarkable potential of LLMs in addressing few-shot vision tasks.
• This work establishes a new direction for few-shot learning by combining linguistic and visual understanding.