S2R-Bench: A Sim-to-Real Evaluation Benchmark for Autonomous Driving

Area of Science:

• Autonomous Driving Systems
• Computer Vision
• Robotics

Background:

• Perception algorithms are critical for autonomous driving safety but lack robustness.
• Existing benchmarks fail to replicate real-world conditions like extreme weather and sensor anomalies.
• Evaluating perception algorithm reliability is an emerging challenge.

Purpose of the Study:

• To propose a novel Sim-to-Real Evaluation Benchmark (S2R-Bench) for autonomous driving.
• To address the limitations of purely simulated benchmarks.
• To foster research into more robust perception models for autonomous vehicles.

Main Methods:

• Collected diverse sensor anomaly data across various real-world road and weather conditions.
• Developed a benchmark dataset encompassing varied lighting intensities and time periods.
• Compared real-world data with simulated data to validate reliability.

Main Results:

• The S2R-Bench is the first corruption robustness dataset based on real-world scenarios.
• Demonstrated the reliability of the collected real-world data for evaluation.
• Highlighted the discrepancy between simulated and real-world performance.

Conclusions:

• The S2R-Bench provides a realistic evaluation for autonomous driving perception systems.
• The dataset facilitates the development of more robust perception algorithms.
• This work is crucial for advancing the safety and reliability of autonomous driving technology.