HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking

Area of Science:

• Computer Vision
• Robotics
• Sensor Fusion

Background:

• RGB cameras provide high spatial resolution, while event cameras excel in temporal resolution and high dynamic range (HDR).
• Combining these sensors offers potential for enhanced object tracking in complex environments.
• Spatio-temporal asymmetries between RGB and event data hinder effective multimodal integration.

Purpose of the Study:

• To develop a multimodal distillation framework, hierarchical asymmetric distillation (HAD), to address spatio-temporal asymmetries.
• To improve object tracking in challenging scenarios by effectively fusing RGB and event camera data.
• To maintain efficiency and compactness of the student network during cross-modal transfer.

Main Methods:

• Proposed hierarchical asymmetric distillation (HAD) framework.
• Employed a hierarchical alignment strategy for cross-modal transfer.
• Utilized multimodal distillation to model and alleviate spatio-temporal asymmetries.

Main Results:

• HAD consistently outperformed state-of-the-art methods in object tracking tasks.
• Comprehensive ablation studies validated the effectiveness of individual HAD components.
• Demonstrated significant improvements in challenging scenarios like high-speed motion and HDR conditions.

Conclusions:

• Hierarchical asymmetric distillation (HAD) offers an effective solution for fusing RGB and event camera data.
• The proposed method successfully mitigates spatio-temporal asymmetries, leading to superior object tracking.
• HAD provides a robust and efficient approach for multimodal sensor fusion in computer vision applications.