Enhancing zero-shot scene recognition through semantic autoencoders and visual relation transfer

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Zero-shot learning (ZSL) aims to recognize images from classes not seen during training.
• Traditional ZSL methods struggle with scene images due to significant intra-class variations.
• Existing approaches often focus on visual-semantic or seen-unseen semantic relationships, yielding suboptimal performance for scene recognition.

Purpose of the Study:

• To develop a novel approach for zero-shot scene image recognition that overcomes limitations of existing methods.
• To improve the recognition performance for unseen classes in complex scene datasets.
• To effectively bridge the domain gap between visual and semantic spaces in ZSL.

Main Methods:

• Proposed a novel approach termed SAEVRT, combining semantic autoencoders (SAEs) and visual relation transfer (VRT).
• Learned two SAEs for seen and unseen scene classes to mitigate domain shift between visual and semantic spaces.
• Developed an interpretable VRT method to learn effective unseen semantic vectors, addressing the lower efficacy of semantic vectors compared to visual features for scene images.

Main Results:

• The SAEVRT method achieved superior performance across four benchmark scene datasets.
• Recognition accuracies reached 63.77% on Scene15, 67.75% on MIT67, 58.68% on UCM21, and 53.26% on NWPU45.
• The unified framework effectively exploited both visual-semantic and seen-unseen relationships.

Conclusions:

• The proposed SAEVRT method significantly advances zero-shot scene image recognition.
• Combining SAEs and VRT offers a robust solution for handling large intra-class variations in scene images.
• The approach demonstrates the potential for more accurate and reliable recognition of unseen visual categories.