Representation learning of crash narrative using natural language processing models.

Area of Science:

• Traffic Safety Research
• Automated Driving Systems (ADS)
• Natural Language Processing (NLP)
• Statistical Learning

Background:

• Understanding crash scenarios is crucial for traffic safety and evaluating automated driving systems (ADS).
• Existing methods often rely on structured data, potentially missing nuances in narrative descriptions.
• A novel framework is needed to effectively process unstructured crash narratives.

Purpose of the Study:

• To develop and validate a framework for constructing a continuous crash space from unstructured narrative text.
• To leverage large language models (LLMs) and statistical learning for enhanced feature extraction from crash data.
• To improve the analysis of crash scenarios, identification of patterns, and evaluation of safety systems.

Main Methods:

• Utilizing Sentence-BERT for embedding crash narratives into a high-dimensional semantic space.
• Applying Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP) for dimensionality reduction.
• Employing K-means clustering with TF-IDF and Gaussian density estimation for crash characterization and outlier detection.

Main Results:

• The framework was applied to 12,741 crashes from the Crash Investigation Sampling System (CISS).
• The resulting crash space exhibited clear structural organization, with similar crashes clustering together.
• Distinct crash types formed separate regions, and rare events were identified as low-density outliers, visualized in 3D.

Conclusions:

• The proposed approach effectively represents diverse crash scenarios using LLMs, enhancing feature extraction beyond structured variables.
• The developed crash space facilitates intuitive visualization, grouping of similar scenarios, and identification of safety-critical corner cases.
• This framework serves as a valuable tool for traffic safety analysis, policy development, and ADS evaluation.