FIT: Enhancing multimodal knowledge graph completion via fine-grained interaction and TriConvTransformer

Area of Science:

• Artificial Intelligence
• Data Science
• Machine Learning

Background:

• Multi-modal Knowledge Graphs (MMKGs) integrate diverse data types like text, audio, and images.
• MMKGs often face data incompleteness, hindering their utility.
• Existing Multi-modal Knowledge Graph Completion (MMKGC) methods struggle with fine-grained inter-modal and entity-relation interactions.

Purpose of the Study:

• To address data incompleteness in MMKGs by enhancing MMKGC.
• To improve fine-grained interactions between modalities and between entities and relations.
• To develop a novel framework for more effective MMKGC.

Main Methods:

• Proposed a framework named FIT (enhancing MMKGC via Fine-Grained Interaction and TriConvTransformer).
• Introduced Fine-Grained Modal Hierarchical Interaction (FMHI) to obtain and fuse multi-modal embeddings.
• Developed the TriConvTransformer decoder for deep entity-relation interactions.
• Incorporated Cross-Modal Self-Attention Contrastive Learning (CM-SACL) and Adaptive Loss Interaction (ALI) for effective multi-modal fusion.

Main Results:

• The proposed FIT framework significantly enhances MMKGC performance.
• FIT demonstrates superior results compared to the latest state-of-the-art models on standard MMKGC benchmarks.
• The method effectively leverages fine-grained interactions and multi-modal fusion.

Conclusions:

• The FIT framework offers a promising solution for Multi-modal Knowledge Graph Completion.
• Enhanced inter-modal and entity-relation interactions are crucial for MMKGC.
• The proposed mechanisms effectively fuse multi-modal information, improving prediction accuracy.