Category-Based Deep CCA for Fine-Grained Venue Discovery From Multimodal Data.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Discovering venues from photographs is crucial for visual context-aware applications.
• Existing methods struggle with complex, user-generated images of real-world venues.
• There's a need for fine-grained venue discovery using heterogeneous social multimodal data.

Purpose of the Study:

• To propose a novel deep learning model for fine-grained venue discovery.
• To enable both exact venue search and group venue search using cross-modal correlations.
• To address the challenge of limited visual information in user-generated venue photographs.

Main Methods:

• Developed a category-based deep canonical correlation analysis (CCA) model.
• Projected data from different modalities (images, text) into a shared space using deep networks.
• Jointly optimized pairwise and category-based correlations for venue search tasks.
• Increased the number of photographs per venue during training to capture richer venue aspects.

Main Results:

• The proposed method demonstrates feasibility on a newly built venue-aware multimodal dataset.
• Experimental results confirm the model's effectiveness in cross-modal retrieval between images and text.
• The method outperforms existing state-of-the-art approaches on a publicly available dataset.

Conclusions:

• The category-based deep CCA model effectively performs fine-grained venue discovery from multimodal social data.
• The approach successfully integrates visual and textual information for accurate venue identification and categorization.
• This work advances the capabilities of visual context-aware applications in understanding real-world venues.