Cross-Modal Object Detection Based on a Knowledge Update.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Object detection methods primarily use visual data, limiting their ability to capture high-level semantic information.
• Existing models face limitations due to the sole reliance on visual cues, hindering comprehensive object understanding.

Purpose of the Study:

• To propose a knowledge update-based multimodal object recognition model.
• To overcome limitations of existing methods by integrating multi-source information for enhanced object detection.

Main Methods:

• Utilizing Faster R-CNN for image regionalization.
• Employing a transformer-based multimodal encoder for visual and textual feature encoding.
• Introducing a graph convolutional network (GCN) for relational network inference and a knowledge update module for enhanced relationship expression.

Main Results:

• The model effectively learns accurate relationships between objects in different image regions.
• Experimental results demonstrate the effectiveness of the knowledge update module.
• The proposed model exhibits independent reasoning capabilities.

Conclusions:

• The developed multimodal object recognition model successfully integrates visual and textual information.
• The knowledge update module significantly enhances the model's ability to express region-based relationships.
• The approach offers improved object detection accuracy and reasoning abilities in computer vision.