VLA-MP: A Vision-Language-Action Framework for Multimodal Perception and Physics-Constrained Action Generation in...

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Autonomous driving systems struggle with complex environments, requiring advanced perception, reasoning, and planning.
• Current end-to-end approaches face limitations in robustness and physical feasibility.

Purpose of the Study:

• To introduce VLA-MP, a unified vision-language-action framework for autonomous driving.
• To enhance the integration of multimodal perception, language understanding, and physics-informed action generation.

Main Methods:

• VLA-MP integrates multimodal Bird's-Eye View (BEV) perception, vision-language alignment, and a GRU-bicycle dynamics cascade adapter.
• It processes RGB images and LiDAR for environmental representation, aligns scene features with natural language via a cross-modal projector and LLM.
• High-level semantic states are converted into physically consistent trajectories.

Main Results:

• VLA-MP achieved top scores on the LangAuto benchmark series (44.3, 63.5, 78.4).
• High infraction scores (0.89-0.95) indicate safe driving performance.
• Outperformed existing VLA methods like LMDrive and AD-H.

Conclusions:

• VLA-MP demonstrates effective language-conditioned driving, adaptability, and safety prioritization.
• Combining multimodal perception, language reasoning, and physics-aware adapters is promising for autonomous driving.
• The framework offers a robust and interpretable approach to autonomous navigation.