Knowledge graph embedding by fusing multimodal content via cross-modal learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Knowledge graph embedding (KGE) traditionally relies on structural information from triples.
• Existing KGE methods often overlook rich entity and relation content, such as text descriptions and images.
• Current multimodal KGE approaches struggle with data heterogeneity and cross-modal correlations.

Purpose of the Study:

• To propose a novel multi-modal content fusion (MMCF) model for enhanced knowledge graph embedding.
• To effectively fuse heterogeneous multimodal data (text, images, structure) for KGE.
• To improve the representation learning of entities and relations by incorporating multimodal information.

Main Methods:

• Developed a cross-modal correlation learning component to fuse intra-modal and inter-modal data.
• Employed a gating network to integrate fused multimodal content with structural features.
• Enhanced relation embeddings by fusing features from associated head and tail entities.

Main Results:

• The proposed MMCF model demonstrated superior performance in link prediction tasks.
• Significant improvements were observed across three benchmark datasets (FB-IMG, WN18RR, FB15k-237).
• MMCF outperformed state-of-the-art baseline methods in most evaluation metrics.

Conclusions:

• The MMCF model effectively addresses the limitations of existing KGE approaches by leveraging multimodal content.
• The proposed cross-modal correlation learning and fusion strategy enhances the quality of knowledge graph embeddings.
• The findings highlight the potential of multimodal fusion for advancing knowledge graph representation learning.