Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision-Language-Action Models for Robotic...

Area of Science:

• Robotics
• Artificial Intelligence
• Sensor Fusion

Background:

• Vision-Language-Action (VLA) models integrate language, vision, and control for general-purpose robotic policies.
• Existing research often overlooks sensor modality selection, fusion, and their impact on robotic manipulation.
• Real-world robotic performance and safety are critically dependent on sensor data processing.

Purpose of the Study:

• To address the gap in understanding sensor modalities within VLA models by adopting a sensor-fusion-action pipeline framework.
• To systematically analyze sensor modalities, data collection, and fusion techniques for robotic manipulation.
• To advocate for a sensor-centric evaluation framework for next-generation Physical AI.

Main Methods:

• Systematic taxonomy and comparison of major sensor modalities (RGB, depth, tactile, force/torque, proprioception, IMU, multi-spectral/thermal, event-based vision).
• Review of teleoperation, human video, and simulation-based data collection pipelines and datasets.
• Analysis of multi-modal design space including fusion techniques (early, late, cross-attention, token-level) and action representations.

Main Results:

• Identified key characteristics, failure modes, and deployment constraints for various sensor modalities.
• Highlighted a bias in current benchmarks towards RGB-centric inputs and single performance metrics.
• Revealed the importance of multi-modal sensor fusion for effective robotic action generation.

Conclusions:

• A sensor-centric perspective is crucial for advancing VLA models beyond current limitations.
• A multidimensional evaluation framework is needed, considering robustness, safety, latency, and efficiency.
• This survey provides a foundation for developing more capable and reliable Physical AI systems.