Multi-Scale Multi-Feature Context Modeling for Scene Recognition in the Semantic Manifold.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Traditional scene recognition relied on two-step inference with intermediate representations, like the semantic manifold (SM).
• SM models patches and images in a semantic probability simplex, but struggles with co-occurring scene categories due to weak supervision.
• Large datasets and Convolutional Neural Networks (CNNs) have largely superseded older methods due to their superior representation learning.

Purpose of the Study:

• To address limitations of the original semantic manifold approach for scene recognition.
• To propose novel, faster discriminative patch representations using neural networks.
• To develop a hybrid architecture combining semantic manifold with multiscale CNNs.

Main Methods:

• Developed discriminative patch representations leveraging neural networks.
• Proposed a hybrid architecture integrating semantic manifold on top of multiscale CNNs.
• Formulated rich context models using Markov random fields to combine multiscale features and spatial relations, optimized via a top-down hierarchical algorithm.

Main Results:

• The proposed hybrid model and discriminative patch representations are significantly faster than original SM's Gaussian mixture models.
• Utilizing Markov random fields effectively captures contextual relations across multiple scales and features.
• The top-down hierarchical optimization algorithm demonstrated superior performance in solving the context modeling problem.

Conclusions:

• The hybrid architecture combining semantic manifold with multiscale CNNs offers a significant improvement over traditional methods.
• Exploiting diverse contextual relations jointly consistently enhances scene recognition accuracy.
• The new approach provides faster computation while maintaining or improving recognition performance.