RAR: Retrieving and Ranking Augmented MLLMs for Visual Recognition

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Contrastive Language-Image Pre-training (CLIP) excels at broad associations but struggles with fine-grained distinctions.
• Multimodal Large Language Models (MLLMs) handle fine-grained classification but degrade with more categories and limited context.
• Existing methods face challenges in few-shot/zero-shot recognition for large, detailed visual vocabularies.

Purpose of the Study:

• To develop a method that synergizes CLIP and MLLMs for enhanced few-shot/zero-shot recognition.
• To address limitations in fine-grained recognition and MLLM performance with increased category numbers.
• To improve accuracy on datasets with extensive and fine-grained visual categories.

Main Methods:

• Introduced RAR (Retrieving And Ranking), an augmented method for MLLMs.
• Established a multi-modal retriever using CLIP to create an explicit memory for categories.
• Implemented a retrieval and ranking process where MLLMs predict based on retrieved memory and context.

Main Results:

• RAR significantly boosts accuracy in vision-language recognition tasks.
• Demonstrated substantial performance improvements on 5 fine-grained visual recognition benchmarks.
• Achieved notable gains on 11 few-shot image recognition datasets and 2 object detection datasets under zero-shot settings.

Conclusions:

• RAR effectively combines the strengths of CLIP and MLLMs for superior fine-grained recognition.
• The method overcomes context window limitations and category number constraints in MLLMs.
• RAR offers a robust solution for few-shot/zero-shot recognition in complex visual datasets.