Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies

Multi-modality visuomotor policies, typically driven by complementary visual sensors such as frame-based RGB cameras and event-based Dynamic Vision Sensors (DVS), have demonstrated strong potential in adverse illumination and high-speed scenarios. Most existing approaches either adopt conventional computer-vision-style feature-level fusion to derive a unified intermediate representation for subsequent value estimation and policy learning, or estimate modality importance separately from a value-level perspective. However, they still face challenges in effectively removing modality-specific task-irrelevant information, leading to suboptimal decisions based on noisy representations. To address this issue, we propose a Decomposed Multi-Modality Representation (DMR) framework that explicitly learns combined task-relevant features (co-features) while separating RGB- and DVS-specific task-irrelevant components. The co-features are trained under task-relevant supervision, while modality-specific components are modeled through reconstruction and InfoNCE-style contrastive objectives to ensure information completeness and discriminability. In addition, existing exploration strategies in reinforcement learning typically encourage transition instability within each modality independently, which often leads to inefficient exploration with limited state-space coverage and low sample efficiency. Therefore, we introduce an intrinsic reward based on Modality Transition Discrepancy (MTD) computed on DMR representations. By quantifying discrepancies between RGB and DVS transition dynamics, MTD promotes exploration of states with meaningful cross-modality differences. Extensive experiments on the CARLA platform and the DDD20 benchmark demonstrate that our method achieves state-of-the-art performance across diverse traffic conditions in both reinforcement learning and imitation learning settings.