SPAN: Learning Similarity Between Scene Graphs and Images With Transformers.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Scene graph generation is crucial for AI applications but lacks effective evaluation metrics.
• Current metrics like Recall@K are sensitive to noise and do not capture overall semantic differences.
• This limits the practical use of generated scene graphs in downstream tasks.

Purpose of the Study:

• To propose the first framework, SPAN (Scene graPh-imAge coNtrastive learning), for learning similarity between scene graphs and images.
• To introduce a new evaluation metric, R-Precision, for scene graph generation.
• To establish new benchmarks for evaluating scene graph-image similarity.

Main Methods:

• Developed a graph Transformer and an image Transformer to align scene graphs and images in a shared latent space.
• Introduced a novel graph serialization technique to convert scene graphs into sequences with structural encodings.
• Proposed R-Precision, an image retrieval accuracy metric, for scene graph generation evaluation.

Main Results:

• The SPAN framework effectively measures scene graph and image similarity.
• R-Precision demonstrates superior performance as an evaluation metric for scene graph generation.
• New benchmarks were established on Visual Genome and Open Images datasets, validating SPAN's effectiveness.

Conclusions:

• SPAN provides a robust method for learning scene graph-image similarity, addressing limitations of existing metrics.
• The proposed R-Precision metric offers a more reliable evaluation for scene graph generation.
• SPAN shows significant potential as a scene graph encoder for various AI applications.