Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Multimodal transformers are effective for visual grounding, aligning image and text data.
• Existing encoder-only frameworks like TransVG face computational challenges due to quadratic time complexity in self-attention.
• High spatial redundancy in images presents an opportunity for optimization.

Purpose of the Study:

• To develop a computationally efficient multimodal transformer for visual grounding.
• To address the limitations of existing encoder-only frameworks by reducing computational overhead.
• To improve the accuracy and efficiency trade-off in visual grounding tasks.

Main Methods:

• Proposed a Dynamic Multimodal Detection Transformer (Dynamic MDETR) architecture, decoupling grounding into encoding and decoding phases.
• Devised a dynamic multimodal transformer decoder utilizing a 2D adaptive sampling module and a text-guided decoding module.
• The decoder adaptively samples informative image patches and refines object localization through cross-attention between image and text features.

Main Results:

• Dynamic MDETR achieved competitive accuracy-accuracy trade-offs, reducing computational costs by approximately 44% GFLOPs with only 9% feature points in the decoder.
• Outperformed TransVG (ResNet-101) using fewer parameters (ResNet-50) with marginal extra computational cost.
• Developed the first one-stage CLIP-empowered visual grounding framework, achieving state-of-the-art performance and demonstrating strong generalization.

Conclusions:

• The proposed Dynamic MDETR offers a significant improvement in computational efficiency for visual grounding without sacrificing accuracy.
• The dynamic decoder effectively exploits image sparsity to accelerate the grounding process.
• The framework shows promising scalability and generalization capabilities, setting a new state-of-the-art.