Semantic-Decoupled and Knowledge-Shared Probabilistic Mapping Network for Multi-Grained Cross-Modal Retrieval

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Cross-modal retrieval is vital for understanding semantic links in multimodal data.
• Existing methods struggle with semantic ambiguity and generalizing from limited samples.

Purpose of the Study:

• To propose a novel Semantic-Decoupled and Knowledge-Shared Probabilistic Mapping Network (SKPMN) for enhanced cross-modal retrieval.
• To address challenges in semantic ambiguity and sparse sample generalization.

Main Methods:

• Developed a Semantic Decoupling and Distinction (SDD) module for relevance-driven representations.
• Introduced a Deep Probability Mapping (DPM) module to map features into probabilistic distributions, capturing uncertainty.
• Integrated an Attention Probabilistic Mapping (APM) module for knowledge transfer and distinction.
• Implemented a multi-grained alignment strategy combining fine-grained and global alignment.
• Utilized a channel resource allocation technique with Joint Source-Channel Coding (JSCC) for efficient visual feature transmission in constrained environments.

Main Results:

• SKPMN demonstrated superior retrieval accuracy on benchmark datasets.
• The model effectively handles semantic similarities and uncertainties in relationships.
• Enhanced generalization capabilities for sparse and ambiguous samples were achieved.
• Improved efficiency in visual feature transmission under resource constraints.

Conclusions:

• The proposed SKPMN effectively overcomes limitations in current cross-modal retrieval approaches.
• The network architecture and strategies significantly improve retrieval performance and generalization.
• The resource allocation technique enhances practical applicability in real-world, constrained scenarios.