Co-Training for Deep Object Detection: Comparing Single-Modal and Multi-Modal Approaches.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Accurate Convolutional Neural Networks (CNNs) require extensive labeled data, which is costly and time-consuming to acquire.
• Domain shifts between image sensors exacerbate the data labeling challenge, potentially necessitating sensor-specific labeling.
• Semi-Supervised Learning (SSL) methods offer a potential solution to reduce reliance on fully labeled datasets.

Purpose of the Study:

• To evaluate the effectiveness of co-training, an SSL technique, for generating self-labeled object bounding boxes (BBs) to train deep object detectors.
• To assess the performance of multi-modal co-training using appearance (RGB) and estimated depth (D) image data.
• To compare multi-modal co-training against single-modal (appearance-based) co-training.

Main Methods:

• Implemented co-training, a semi-supervised learning approach, to generate self-labeled ground truth (GT) for object detection.
• Utilized multi-modal data fusion, combining RGB appearance and estimated depth (D) views of images.
• Compared performance against single-modal co-training (RGB only) in standard and domain-shifted settings.

Main Results:

• Multi-modal co-training outperformed single-modal co-training in both standard SSL settings and virtual-to-real domain shift scenarios.
• In virtual-to-real domain shift, Generative Adversarial Network (GAN)-based domain translation enabled both co-training modalities to achieve comparable performance.
• The effectiveness holds even with an off-the-shelf depth estimation model on translated images.

Conclusions:

• Multi-modal co-training is a promising approach to mitigate the data labeling bottleneck in training CNNs for object detection.
• Leveraging both appearance and depth information enhances model robustness, particularly under domain shifts.
• GAN-based domain translation can further improve performance in cross-domain scenarios.