From Concept to Representation: Modeling Driving Capability and Task Demand with a Multimodal Large Language Model

Area of Science:

• Intelligent transportation systems
• Artificial intelligence in automotive safety
• Human-computer interaction in driving

Background:

• Driving safety relies on the balance between task demand and driver capability.
• Current methods lack a unified, quantifiable approach to assess this interplay.
• Existing frameworks struggle to predict potential capability shortfalls before unsafe driving occurs.

Purpose of the Study:

• To develop a novel framework for quantifying driving task demand and capability using multimodal data.
• To create an interpretable measurement of task difficulty for proactive risk assessment.
• To enable early detection of driver capability degradation and potential safety risks.

Main Methods:

• Utilized a multimodal large language model (LLM) integrating scene images, maneuver descriptions, control inputs, and traffic states.
• Transformed heterogeneous driving signals into low-dimensional embeddings for task demand and driving capability.
• Projected embeddings into a shared latent space for interpretable task difficulty measurement.
• Employed a customized BLIP 2 backbone, fine-tuned on diverse simulated driving scenarios.

Main Results:

• The framework successfully quantifies task demand and driving capability from multimodal inputs.
• An interpretable measurement of task difficulty was generated, correlating with potential capability shortfalls.
• The model demonstrated consistency within tasks and captured impairment-related capability degradation.
• The framework showed effective transferability to real-world motorway data without retraining.

Conclusions:

• The proposed LLM-based framework offers a concise and effective method for proactive risk assessment in intelligent vehicles.
• It provides an explainable measurement of driving difficulty, crucial for enhancing vehicle safety.
• This approach paves the way for more sophisticated safety systems that anticipate and mitigate risks before they materialize.