Graph Embedding Contrastive Multi-Modal Representation Learning for Clustering.

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• Multi-modal clustering (MMC) methods struggle with unified objectives for inter- and intra-modality consistency.
• Existing deep learning-based MMC approaches often fail to handle out-of-sample data effectively.

Purpose of the Study:

• To propose a novel network, Graph Embedding Contrastive Multi-modal Clustering (GECMC), addressing limitations in current deep learning-based MMC.
• To integrate representation learning and multi-modal clustering into a unified framework.
• To enable handling of out-of-sample data in multi-modal clustering.

Main Methods:

• GECMC employs a co-training framework where representation learning and clustering interact and evolve.
• A contrastive loss is designed using pseudo-labels to enhance consistency across modalities.
• The network maximizes intra-cluster representation similarity while minimizing inter-cluster similarity at both inter- and intra-modality levels.
• A clustering layer with learnable centroids allows for handling out-of-sample data.

Main Results:

• GECMC demonstrates superior performance compared to 14 existing methods across four challenging datasets.
• The proposed method effectively learns clustering labels for given samples.
• GECMC successfully handles out-of-sample data, a significant improvement over prior work.

Conclusions:

• GECMC offers an effective solution for multi-modal clustering by unifying representation learning and clustering.
• The network's ability to manage inter- and intra-modality consistency and handle out-of-sample data marks a significant advancement.
• The proposed approach provides a robust and scalable solution for complex multi-modal clustering tasks.