Exploring Effective Factors for Improving Visual In-Context Learning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• In-context learning (ICL) enables models to grasp new tasks from few demonstrations without fine-tuning.
• While established in Natural Language Processing (NLP), ICL is an emerging research area in computer vision.
• Understanding factors influencing visual ICL performance is crucial for advancing the field.

Purpose of the Study:

• To identify key factors affecting visual in-context learning performance.
• To propose a novel framework, prompt-SelF, to enhance visual ICL.
• To demonstrate the effectiveness of prompt-SelF on computer vision tasks.

Main Methods:

• Investigated prompt selection and prompt fusion as critical determinants of visual ICL performance.
• Developed prompt-SelF, a framework employing pixel-level retrieval for prompt selection.
• Utilized diverse prompt fusion techniques and ensembled predictions for improved accuracy.

Main Results:

• Prompt selection and prompt fusion significantly impact visual ICL inference.
• The prompt-SelF framework demonstrated superior performance on single-object segmentation and detection tasks.
• prompt-SelF achieved state-of-the-art results, outperforming meta-learning approaches in 1-shot segmentation.

Conclusions:

• Prompt selection and fusion are pivotal for effective visual in-context learning.
• The prompt-SelF framework offers a significant advancement in visual ICL.
• Visual ICL shows great potential for future computer vision applications.