Human-like scene graph generation and evaluation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Existing image scene graph generation methods produce overly dense graphs, hindering relevance assessment and downstream applications.
• Current methods lack the ability to effectively capture the essential elements of a scene as perceived by humans.

Purpose of the Study:

• To develop a novel method for generating image scene graphs that align with human perception of importance.
• To improve the effectiveness of scene graphs for tasks such as cross-modal retrieval and data mining.

Main Methods:

• A new scene graph generation method guided by human-drafted image captions during training.
• Utilizing specifically designed loss functions to incorporate human relevance into the graph generation process.
• Evaluating generated scene graphs against human-created ground-truth data using metrics like recall, precision, and graph edit distance.

Main Results:

• The proposed method, incorporating human-caption guidance and novel loss functions, demonstrates improved performance, particularly in graph edit distance.
• Benchmarking against existing models shows significant gains when the new loss functions are applied.
• Analysis indicates that accurately recognizing less important background objects and their relationships is vital for human-like scene graph generation.

Conclusions:

• The novel scene graph generation method effectively produces graphs that reflect human-perceived importance.
• Guidance from human captions and tailored loss functions are crucial for generating relevant and human-like scene graphs.
• Future work should focus on the accurate recognition of all scene elements, including background details, for enhanced graph generation.