Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Transfer learning is crucial for adapting neural networks to new tasks.
• Scene classification accuracy is limited by holistic analysis alone.

Purpose of the Study:

• To develop a novel method for scene classification by transferring object recognition capabilities.
• To improve scene classification performance by effectively encoding semantic information.

Main Methods:

• A Bag-of-Semantics (BoS) representation was created using object recognition CNNs.
• Fisher Vectors (FV) were adapted for encoding BoS, focusing on the natural parameter space of multinomial distributions.
• A Mixture of Factor Analyzers (MFA) model was employed for sophisticated semantic feature covariance modeling, implemented as MFAFSNet for end-to-end training.

Main Results:

• The proposed MFAFSNet achieved state-of-the-art transfer performance in scene classification.
• The method surpassed the performance of CNNs explicitly trained for scene classification on large datasets.
• Combining local object semantics with holistic analysis yielded significant complementary gains.

Conclusions:

• Modeling local object semantics is as important as holistic scene analysis for superior scene classification.
• The proposed Bag-of-Semantics approach with MFA Fisher Scores offers a powerful and complementary strategy for scene classification.
• This research opens new avenues for leveraging object recognition models in scene understanding tasks.