Pre-training Model Based on Parallel Cross-Modality Fusion Layer.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual Question Answering (VQA) integrates computer vision and natural language processing.
• Understanding the alignment between visual concepts and linguistic semantics is crucial for VQA.

Purpose of the Study:

• To propose a novel Pre-training Model Based on Parallel Cross-Modality Fusion Layer (P-PCFL).
• To learn fine-grained relationships between vision and language for improved VQA.
• To enhance the reasoning and universality of VQA models.

Main Methods:

• The P-PCFL model utilizes three core Transformer-based Encoders: Object, Language, and Parallel Cross-Modality Fusion.
• Four pre-training tasks were employed: Cross-Modality Mask Language Modeling, Cross-Modality Mask Region Modeling, Image-Text Matching, and Image-Text Q&A.
• The model was pre-trained to learn Intra-modality and Inter-modality relationships.

Main Results:

• Pre-trained P-PCFL model demonstrated significant effectiveness on the VQA v2.0 dataset after fine-tuning.
• Ablation experiments and attention visualization confirmed the model's efficacy.
• The approach successfully improved the model's ability to understand vision-language connections.

Conclusions:

• The proposed P-PCFL model is effective for learning fine-grained vision-language relationships in VQA.
• Pre-training with the P-PCFL approach enhances VQA model performance and generalization.
• The study validates the P-PCFL model's contribution to advancing VQA research.