Query by Example: Semantic Traffic Scene Retrieval Using LLM-Based Scene Graph Representation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Robotics

Background:

• Autonomous driving systems require efficient retrieval of specific traffic scenes from vast datasets.
• Existing methods face challenges with the semantic complexity and varied user needs in traffic scene data.
• The heterogeneity and scale of autonomous driving data pose significant retrieval hurdles.

Purpose of the Study:

• To propose a novel traffic scene retrieval method for autonomous driving.
• To leverage Visual-Large Language Models (VLMs) for structured Road Scene Graph (RSG) representation.
• To enable flexible and semantically rich querying of traffic scenes.

Main Methods:

• Utilized VLMs to generate structured RSGs from video data, capturing object relationships and semantic attributes.
• Developed an extensible set of scene attributes and a graph-based description for quantifying scene similarity.
• Created the RSG-LLM benchmark dataset with 1000 traffic scenes, descriptions, and RSGs for LLM evaluation.

Main Results:

• The proposed method effectively retrieves semantically similar traffic scenes from large databases.
• The approach supports diverse query formats, including natural language, images, video clips, and rosbag files.
• Demonstrated the capability of VLMs in generating accurate RSGs for traffic scenes.

Conclusions:

• The VLM-generated RSG approach offers a comprehensive and flexible framework for traffic scene retrieval.
• This method significantly improves the efficiency and accuracy of scene retrieval in autonomous driving.
• The proposed framework facilitates broader applications within autonomous driving systems.