Decomposed Multi-Modality Fusion: Integrating Frames and Events for Efficient Visuomotor Policies.

Area of Science:

• Robotics and Computer Vision
• Reinforcement Learning

Background:

• Multi-modality visuomotor policies leverage sensors like RGB cameras and Dynamic Vision Sensors (DVS) for challenging conditions.
• Existing methods struggle to filter task-irrelevant information, leading to suboptimal performance.

Purpose of the Study:

• To propose a Decomposed Multi-Modality Representation (DMR) framework for effective fusion of multi-modal sensor data.
• To enhance reinforcement learning exploration using a novel Modality Transition Discrepancy (MTD) reward.

Main Methods:

• Developed the DMR framework to learn joint task-relevant features (co-features) while isolating modality-specific irrelevant components.
• Employed reconstruction and contrastive objectives for modality-specific components.
• Introduced MTD reward based on DMR representations to guide exploration.

Main Results:

• Achieved state-of-the-art performance in both reinforcement learning and imitation learning settings.
• Demonstrated superior results on the CARLA platform and DDD20 benchmark.
• Showcased improved decision-making and exploration efficiency.

Conclusions:

• The proposed DMR framework effectively addresses challenges in multi-modality fusion for visuomotor policies.
• MTD-based exploration significantly improves state-space coverage and sample efficiency.
• The approach shows strong potential for real-world autonomous driving applications.