Critical voxel learning with vision transformer and derivation of logical AV safety assessment scenarios

Safety assessment is an active research subject for autonomous vehicles (AVs) that have emerged as a new mode of mobility. In particular, scenario-based safety assessments have garnered significant attention. AVs can be tested on how they safely avoid hypothetical situations leading to accidents. However, scenarios written by humans based on their expert knowledge and experience may only partially reflect real-world situations. Instead, we are keen on a different technique of extracting statistically significant and more detailed scenarios from sensor data captured during the critical moments when AVs become vulnerable to potential accidents. Specifically, we first render the three-dimensional space around an AV with fixed-sized voxels. Then, we modeled the aggregate kinetics of the objects in each voxel detected by 3D-LiDAR sensors mounted on real test AVs. The Vision Transformer we used to model the kinetics helped us quickly pinpoint critical voxels containing objects that threatened the AV's safety. We traced the trajectory of the critical voxels on a visual attention map to describe in detail how AVs become vulnerable to accidents according to the logical scenario format defined by the PEGASUS Project. We tested our novel method with 250 h of 3D-LiDAR recordings capturing critical moments. We devised an inference model that detected critical situations with an F1-score of 98.26%. For each type of scenario, our model consistently identified the critical objects and their tendency to influence AVs. Given the evaluation results, we can ensure that our data-driven approach yields an AV safety assessment scenario with high representativeness, coverage, expansion, and computational feasibility.