C3Net: A cross-modal collaborative calibration of features for object detection using frames and events.

Area of Science:

• Computer Vision
• Machine Learning
• Sensor Fusion

Background:

• Fusing RGB frames and event streams for object detection is challenging due to data heterogeneity.
• Existing methods often suffer from suboptimal fusion and statistical disparities between modalities.

Purpose of the Study:

• To develop a novel framework, C3Net, that overcomes the limitations of direct feature merging for RGB and event stream fusion.
• To improve object detection performance by addressing the inherent heterogeneity and statistical disparities between RGB and event data.

Main Methods:

• Introduced C3Net, a framework utilizing Collaborative Calibration instead of direct feature merging.
• Developed an Adaptive Balancing Time Surface (ABTS) for motion-robust event representations.
• Implemented a Cross-Modal Feature Collaborative Calibration Module (CM-FCCM) for mutual calibration of RGB and event features.
• Utilized an Adaptive Channel Fusion Module (ACFM) for dynamic, confidence-based modality integration.

Main Results:

• C3Net achieved state-of-the-art performance on benchmark datasets (PKU-DAVIS-SOD, DSEC-MOD, PKU-DDD17-CAR).
• Demonstrated superior ability in leveraging complementary strengths of RGB frames and event streams.
• The proposed modules (ABTS, CM-FCCM, ACFM) effectively mitigate modality discrepancies and enhance feature learning.

Conclusions:

• C3Net offers a paradigm shift in fusing heterogeneous sensor data for object detection.
• The Collaborative Calibration approach significantly improves detection accuracy and robustness.
• C3Net effectively handles varying object velocities and reduces spatial inconsistencies in event data.