Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Open-Vocabulary Object Detection (OVD) aims to detect objects from unseen classes.
• Existing OVD methods struggle with precise localization and fine-grained visual context modeling.
• Multi-modal learning offers potential for improved cross-modal understanding between text and images.

Purpose of the Study:

• To propose a novel multi-modal contextual knowledge distillation framework (MMC-Det) for enhanced OVD.
• To leverage multi-modal masked language modeling (MLM) for explicit localization guidance.
• To improve the modeling of fine-grained region-level visual contexts in object detection.

Main Methods:

• Developed a multi-modal contextual knowledge distillation framework (MMC-Det).
• Employed a teacher fusion transformer trained with a diverse multi-modal masked language modeling (D-MLM) strategy.
• Supervised a student detector using context-aware attention from masked concept words.

Main Results:

• The D-MLM strategy significantly enhances fine-grained region-level visual context modeling.
• The proposed distillation process provides effective contextual guidance for concept-region matching.
• Experimental results demonstrate the effectiveness of the multi-modal context learning strategy in OVD.

Conclusions:

• Multi-modal contextual knowledge distillation is effective for improving open-vocabulary object detection.
• The proposed D-MLM strategy enhances the understanding of visual contexts for better localization.
• MMC-Det offers a promising direction for advancing OVD capabilities.