Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Environmental illusions (shadows, reflections) are overlooked in autonomous driving (AD).
• These illusions pose significant safety risks by distorting visual perception.
• Existing research inadequately addresses the impact of environmental illusions on AD systems.

Purpose of the Study:

• To evaluate the robustness of lane perception in AD systems against environmental illusions.
• To introduce the first benchmark, LanEvil++, for assessing AD lane perception under various illusions.
• To develop and validate a defense mechanism for enhancing AD robustness.

Main Methods:

• Developed LanEvil++, a benchmark with 14 illusion types across 94 high-fidelity 3D scenes using the CARLA simulator.
• Generated a dataset including 90,292 images, 1,596 video clips, and 41,855 visual question-answering pairs.
• Proposed the Multimodal Illusion Defense Approach (MIDA) using hard examples to improve illusion resistance.

Main Results:

• Environmental illusions significantly degrade conventional lane detection (LD) and vision-language model-based systems (ADVLMs).
• LD models showed an average performance drop of 5.27% in Accuracy and 10.49% in F1-score.
• ADVLMs experienced a 2.03% reduction in GPT-score and 0.75% in Language-score; shadows were the most disruptive illusion.
• Closed-loop simulations and real-world case studies confirmed safety-critical failures due to these illusions.

Conclusions:

• Environmental illusions present a critical threat to the safety and reliability of autonomous driving systems.
• The LanEvil++ benchmark and MIDA defense approach offer valuable tools for advancing AD robustness.
• Further research is needed to develop more resilient AD systems capable of handling real-world visual complexities.