Multi-Perspective Cross-Modal Object Encoding for Referring Expression Comprehension.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring Expression Comprehension (REC) is vital for identifying objects in images using text. Existing two-stage REC methods often overlook object context, limiting representation accuracy.
• Current methods rely on pre-trained detectors, neglecting crucial contextual and relational information between objects and their surroundings.

Purpose of the Study:

• To enhance two-stage Referring Expression Comprehension (REC) methods by introducing novel modules.
• To improve the discriminative power of object representations by incorporating contextual and relational information.

Main Methods:

• Proposed two plug-and-adapt modules: expression-guided label representation (ELR) and cross-modal calibrated semantic (CCS) modules.
• ELR module aligns text expression noun phrases with image object labels.
• CCS module integrates object features with neighboring object features from multiple perspectives, preserving intrinsic information and incorporating relational cues.

Main Results:

• The proposed modules significantly strengthen the alignment between image and text representations.
• Experiments on six datasets demonstrate the effectiveness of incorporating prior statistical knowledge.
• The method achieves competitive performance and is compatible with existing two-stage REC approaches.

Conclusions:

• The novel ELR and CCS modules effectively enhance two-stage REC methods by capturing crucial contextual and relational information.
• Integrating prior statistical knowledge and cross-modal calibration leads to more precise object embeddings and improved REC performance.
• The proposed approach offers a versatile enhancement for existing REC frameworks.