MTRAG: Multi-Target Referring and Grounding via Hybrid Semantic-Spatial Integration

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Fine-grained visual referring and grounding are essential for scene understanding and vision-language applications.
• Existing multimodal large language models (MLLMs) struggle with fine-grained multi-target scenarios.

Purpose of the Study:

• To propose MTRAG, a pixel-level framework for multi-target referring and grounding that addresses limitations in current MLLMs.
• To enhance semantic-spatial collaboration for improved performance in complex visual tasks.

Main Methods:

• Introduced Channel Extension Mechanism (CEM) for global and multi-region feature extraction without additional region extractors.
• Developed a grounding branch for pixel-level grounding and a Hybrid Adapter (HA) to fuse semantic and spatial features.
• Curated MTRAG-D dataset and MTR-Bench benchmark for systematic evaluation of multi-target referring.

Main Results:

• MTRAG consistently outperforms strong baselines on both multi-target and single-target referring and grounding tasks.
• The framework maintains competitive performance in image-level captioning.
• Demonstrated effective semantic-spatial alignment through the Hybrid Adapter.

Conclusions:

• MTRAG offers a robust solution for pixel-level multi-target referring and grounding.
• The proposed methods significantly advance the capabilities of MLLMs in fine-grained visual understanding.
• MTRAG provides a valuable benchmark for future research in multi-target visual tasks.