Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Scene understanding is a fundamental challenge in computer vision.
• Parsing scene images into structured object hierarchies and relations requires detailed annotations.
• Existing methods often rely on extensive, manually labeled data.

Purpose of the Study:

• To develop a weakly-supervised deep learning model for scene parsing.
• To automatically discover hierarchical object structures and inter-object relations from images.
• To reduce the need for elaborate manual annotations in scene understanding tasks.

Main Methods:

• A deep architecture combining a Convolutional Neural Network (CNN) for image representation and a Recursive Neural Network (RsNN) for structure discovery.
• Weakly-supervised training using descriptive sentences of training images, decomposed into semantic trees.
• Expectation-Maximization (EM) method for model training, updating CNN and RsNN parameters via backpropagation.

Main Results:

• The proposed model effectively produces meaningful scene configurations.
• Achieved favorable scene labeling results on PASCAL VOC 2012 and SYSU-Scenes benchmarks.
• Demonstrated superior performance compared to state-of-the-art weakly-supervised deep learning methods.

Conclusions:

• Weakly-supervised learning leveraging image-descriptive sentences is effective for scene parsing.
• The proposed CNN-RsNN architecture successfully captures hierarchical object structures and relations.
• The developed SYSU-Scenes dataset advances research in weakly-supervised scene understanding.