TBKIN: Threshold-based explicit selection for enhanced cross-modal semantic alignments.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-language models integrate visual and linguistic data for multi-modal tasks.
• Existing models struggle to minimize interference from irrelevant data, limiting performance.
• Effective semantic alignment between image-text pairs is crucial.

Purpose of the Study:

• To propose a novel vision-language model, the threshold-based knowledge integration network (TBKIN).
• To effectively capture intra-modal and cross-modal knowledge while mitigating extraneous information.
• To enhance semantic alignments and reduce interference in vision-language tasks.

Main Methods:

• TBKIN utilizes unified scene graph structures and advanced masking strategies.
• A fine-tuning strategy based on threshold selection is employed to eliminate noise.
• The model integrates intra-modal and cross-modal knowledge.

Main Results:

• Achieved state-of-the-art accuracy of 73.90% on the VQA 2.0 dataset.
• Achieved state-of-the-art accuracy of 84.60% on the RefCOCO dataset.
• Demonstrated robustness and superior performance across four benchmark datasets.

Conclusions:

• TBKIN effectively reduces interference while enhancing semantic alignments in vision-language tasks.
• The model shows significant potential for advancing multi-modal learning.
• Offers a practical and effective solution for real-world applications.