Language-aware multiple datasets detection pretraining for DETRs.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Scaling annotated datasets for object detection is hindered by high labor costs.
• Numerous isolated, field-specific datasets exist, offering potential for joint pretraining.
• Leveraging diverse datasets can significantly enhance data volume and model robustness.

Purpose of the Study:

• To propose a framework (METR) for pretraining DETR-like object detectors using multiple datasets without manual label space integration.
• To convert multi-classification object detection into binary classification using pre-trained language models.
• To improve data volume and diversity for more effective object detection model pretraining.

Main Methods:

• Developed METR, a framework for joint pretraining of object detectors on aggregated datasets.
• Introduced a category extraction module using language embeddings to assign categories to queries.
• Implemented a class-wise bipartite matching strategy to align ground truths with category-assigned queries.

Main Results:

• METR demonstrated strong performance in both multi-task joint training and pretrain & finetune paradigms.
• Pretrained models exhibited flexible transferability, boosting performance on various DETR-like detectors.
• Significant performance improvements were observed on the COCO val2017 benchmark.

Conclusions:

• METR offers an effective solution for leveraging multiple datasets in object detector pretraining.
• The proposed method eliminates the need for manual label space integration, simplifying the process.
• METR enhances the performance and transferability of DETR-like object detection models.