Scene Graph Prediction with Limited Labels.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Visual knowledge bases are crucial for AI applications like visual question answering.
• Current scene graph models struggle with sparse relationships and require extensive labeled data.
• Human annotation is costly, and text-based methods don't suit visual data.

Purpose of the Study:

• To develop a semi-supervised method for labeling visual relationships in images with limited labeled examples.
• To improve the training of scene graph models for better visual understanding.
• To address the challenge of data scarcity in visual relationship prediction.

Main Methods:

• A semi-supervised approach using image-agnostic features and noisy heuristics.
• A factor graph-based generative model to aggregate heuristic outputs.
• Training state-of-the-art scene graph models with generated data from as few as 10 labeled examples per relationship.

Main Results:

• Outperformed all baseline approaches on scene graph prediction, achieving 5.16 recall@100 for PREDCLS.
• Demonstrated success in limited label settings, outperforming transfer learning.
• Introduced a relationship complexity metric that predicts method success (R² = 0.778).

Conclusions:

• The proposed semi-supervised method effectively generates training data for scene graph models with minimal supervision.
• This approach significantly improves visual relationship prediction accuracy.
• The method offers a cost-effective and efficient solution for building comprehensive visual knowledge bases.