Multimodal Learning of Social Image Representation by Exploiting Social Relations

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Social image analysis benefits from representation learning, but existing methods often overlook multimodal content and social relations.
• Suboptimal embedding can result from solely modeling content information in social images.

Purpose of the Study:

• To propose a novel multimodal representation learning model for social images that integrates both content and social relations.
• To address the challenge of learning unified representations from multimodal data with complex correlations.

Main Methods:

• A correlational multimodal variational autoencoder (CMVAE) was developed to learn a unified representation by encoding common and private information across modalities.
• A triplet network was employed to incorporate social relations among images into the representation learning process.
• A joint embedding model combined social relations with multimodal content for enhanced representation.

Main Results:

• The proposed CMVAE method demonstrated effectiveness on four datasets for multilabel classification and cross-modal retrieval tasks.
• Experimental results showed significant performance improvements over state-of-the-art multimodal representation learning techniques.

Conclusions:

• The CMVAE model successfully integrates multimodal content and social relations for superior social image representation learning.
• This approach offers a significant advancement in handling the complexities of social image data for AI applications.