Foundation Models based Scene Graph Generation

Scene Graph Generation (SGG) is an essential task at the interface of computer vision and natural language processing, aimed at creating structured semantic representations of visual scenes. However, traditional paradigms that train from scratch with SGG datasets face persistent issues, including model bias due to long-tail distributions and difficulties with zero-shot triplet recognition due to sparse annotations. Recently, Foundation Models (FMs), pretrained on extensive datasets, have demonstrated remarkable success in various downstream tasks, showcasing their potential in enhancing SGG task. Consequently, we propose a FMs based SGG (FMSGG) framework, leveraging the broad pretraining of FMs to benefit tail relationships and zero-shot triplets. FMSGG addresses three key challenges when incorporating FMs into SGG task: inefficient training and inference, difficulty in distinguishing similar relationships, and dual-bias. Specifically, our method involves isolating relationship embeddings from image features obtained from FMs via relationship prompts, and augmenting text prompts to enhance the distinctiveness of relationship description embeddings. Additionally, we introduce the triplet-aware logit adjustment method to mitigate long-tail bias, and propose adaptive evidential smoothing to specifically alleviate inherent biases in FMs. Experimental results across various evaluation modes and metrics indicate that FMSGG achieves state-of-the-art mean recall rate and zero-shot recall rate, while also providing an optimal balance between mean recall rate and recall rate.