Semantics Disentangling for Cross-Modal Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval faces challenges due to heterogeneous data gaps and inconsistent distributions.
• Existing methods struggle to bridge these gaps by capturing common representations, often entangling semantic and non-semantic features.

Purpose of the Study:

• To disentangle semantic-shared and semantic-unrelated representations in cross-modal data.
• To improve cross-modal retrieval performance by creating purer semantic representations.

Main Methods:

• Developed a novel Semantics Disentangling approach for Cross-Modal Retrieval (SDCMR).
• Utilized a variational auto-encoder to explicitly decouple semantic-shared and semantic-unrelated features.
• Employed a dual adversarial mechanism with a pushing-and-pulling strategy for feature disentanglement.
• Performed reconstruction by exchanging shared semantics to ensure semantic consistency.

Main Results:

• The proposed SDCMR method achieved superior performance on four widely used datasets.
• SDCMR set a new benchmark, outperforming 15 state-of-the-art methods.
• Explicitly disentangling features led to purer semantic representations, enhancing retrieval accuracy.

Conclusions:

• SDCMR effectively addresses the challenges of cross-modal retrieval by disentangling semantic and non-semantic features.
• The method demonstrates significant improvements in performance and establishes a new state-of-the-art.
• This approach offers a promising direction for future research in cross-modal understanding.