New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM...

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Referring Expression Comprehension (REC) is a key cross-modal task for evaluating Multimodal Large Language Models (MLLMs).
• Existing REC datasets often lack controllable difficulty and robust negative examples, limiting model evaluation.
• Fine-grained reasoning across objects, attributes, and relationships is crucial for advanced REC.

Purpose of the Study:

• To address limitations in existing REC datasets and advance the capabilities of MLLMs.
• To introduce a new REC dataset with controllable difficulty and negative samples for rigorous model testing.
• To propose novel methods combining specialist models and MLLMs for improved REC performance.

Main Methods:

• Developed a new REC dataset featuring controllable difficulty levels and fine-grained negative examples.
• Proposed a hybrid approach that routes simple REC tasks to lightweight models and complex tasks to MLLMs.
• Introduced a collaborative method where specialist models propose regions and MLLMs perform final selection.

Main Results:

• Significant performance improvements were observed on the new REC dataset and other benchmarks.
• The proposed methods demonstrated enhanced accuracy and efficiency in vision-language tasks.
• Combining specialist and general-purpose models proved effective for complex real-world applications.

Conclusions:

• The new REC dataset provides a more challenging and realistic benchmark for MLLMs.
• Hybrid and collaborative AI strategies offer a practical and effective approach to advanced REC.
• This work paves the way for more robust and capable vision-language models.