Inter- and Intra-Modal Contrastive Hybrid Learning Framework for Multimodal Abstractive Summarization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Abstractive summarization technologies enable users to read article summaries online.
• Current multimodal analysis methods suffer from a semantic gap between vision and language, neglecting modality heterogeneity and cross-modal correlations.
• This leads to poor quality learned representations in existing models.

Purpose of the Study:

• To propose a novel Inter- and Intra-modal Contrastive Hybrid (ITCH) learning framework.
• To automatically align multimodal information (text and images) and maintain semantic consistency.
• To develop a component adaptable for both supervised and unsupervised learning approaches.

Main Methods:

• Developed a hybrid learning framework integrating inter- and intra-modal contrastive learning.
• Focused on aligning features across different modalities (vision and language).
• Ensured semantic consistency throughout the input and output data flows.

Main Results:

• The proposed ITCH framework demonstrated superior performance compared to existing baselines.
• Experiments were conducted on two public datasets: MMS and MSMO.
• ITCH effectively addressed the limitations of feature aggregation and modality heterogeneity.

Conclusions:

• The ITCH framework offers an effective solution for multimodal abstractive summarization.
• Improved representation learning by addressing the semantic gap and modality heterogeneity.
• ITCH enhances the quality of generated summaries from multimodal inputs.