A multimodal accident scenario database construction method for end-to-end autonomous vehicle testing

End-to-end autonomous driving represents a major developmental direction, with testing and validation requiring a large-scale, multimodal scenario database. Accident reports offer a valuable source for constructing such a database; however, these reports typically rely on textual descriptions to convey scenario information, making it difficult to capture the temporal dynamics essential for end-to-end autonomous vehicle testing. To address this limitation, a method is proposed for constructing a multimodal accident scenario database tailored to the testing of end-to-end autonomous vehicles. First, a novel MCAT-BiLSTM-CRF algorithm is introduced to efficiently and accurately extract scenario information from accident reports, converting unstructured text into structured representations of scenario elements and auxiliary information. Next, an ontology-based framework is developed, and a preliminary textual knowledge base is constructed using knowledge graph. Furthermore, based on the encoding of scenario element combination sequences, a HyCon-Sg-Net algorithm is developed to expand the textual knowledge base. Finally, by applying an enhanced fine-tuning framework to the Open-Sora model, modality transformation from textual knowledge base to accident video database is achieved, resulting in the construction of a multimodal accident scenario database. An experimental analysis was conducted on the constructed multimodal database. The results show that the proposed MCAT-BiLSTM-CRF algorithm effectively extracts accident-related information, and the fine-tuned Open-Sora model is capable of generating high-quality accident videos consistent with textual descriptions. Evaluated on the constructed multimodal database, a representative autonomous driving perception algorithm exhibited a 48-percentage-point reduction in detection success rate, demonstrating the effectiveness of the proposed database in exposing potential perception weaknesses under extreme accident scenarios and its potential for safety- and robustness-oriented evaluation and stress testing.