Reinforcement learning for online testing of autonomous driving systems: a replication and extension study

Area of Science:

• Software Engineering
• Artificial Intelligence
• Autonomous Systems

Background:

• Deep Neural Network (DNN) systems, particularly Autonomous Driving Systems (ADS), require robust online testing.
• Previous studies suggested Reinforcement Learning (RL) combined with many-objective search excels in testing DNN-enabled systems.
• Replication of prior work revealed RL did not outperform random search under identical conditions, suggesting confounding factors.

Purpose of the Study:

• To replicate and extend a prior empirical study on RL for online testing of Autonomous Driving Systems (ADS).
• To investigate and address potential reasons for RL's underperformance in the replication, focusing on reward function design and state-space discretization.
• To develop and evaluate an improved RL agent for more effective online ADS testing.

Main Methods:

• Replication of an existing study comparing Reinforcement Learning (RL), random search, and many-objective search for online testing of an Autonomous Driving System (ADS).
• Modification of the experimental setup to eliminate confounding factors in collision measurement.
• Extension involving the development of a new RL agent designed to handle continuous state spaces and mitigate conflicting reward signals.

Main Results:

• The replication confirmed that RL did not outperform random search when confounding factors were removed.
• The extended study demonstrated that the newly developed RL agent successfully converged to an effective policy.
• The improved RL agent significantly outperformed random search in the online testing of the Autonomous Driving System (ADS).

Conclusions:

• The effectiveness of RL for online ADS testing is sensitive to reward function design and state-space representation.
• Addressing these factors can lead to RL agents that significantly outperform traditional methods like random search.
• Further research is warranted to optimize RL strategies for advanced online testing of autonomous systems.