A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Latent Dirichlet Allocation (LDA) and Deep Boltzmann Machines (DBM) are common for multimodal data. Document Neural Autoregressive Distribution Estimator (DocNADE) excels in text modeling.
• Existing methods for multimodal data, like image annotation, have limitations in capturing joint representations.

Purpose of the Study:

• To adapt and extend the DocNADE model for effective multimodal data processing, specifically for image classification and annotation.
• To introduce a supervised extension (SupDocNADE) for enhanced discriminative topic features and a deep extension for improved performance.

Main Methods:

• Developed SupDocNADE, a supervised DocNADE variant, to learn joint representations from visual words, annotation words, and class labels.
• Proposed a deep extension of SupDocNADE with an efficient training method for multimodal data.
• Evaluated models on LabelMe, UIUC-Sports, and MIR Flickr datasets.

Main Results:

• SupDocNADE favorably compares to existing topic models on benchmark datasets.
• The deep extension of the model outperforms its shallow version.
• The proposed deep model achieves state-of-the-art performance on the MIR Flickr dataset for multimodal tasks.

Conclusions:

• DocNADE can be effectively extended for multimodal data, outperforming traditional topic models.
• Supervised and deep extensions of DocNADE significantly improve performance in image classification and annotation.
• The developed deep model represents a new state-of-the-art for multimodal information retrieval.