Shadow-DETR: Alleviating matching conflicts through shadow queries

Leveraging the end-to-end detection capability enabled by one-to-one matching, DETR has achieved state-of-the-art performance in simplified pipelines. However, the one-to-one matching mechanism also introduces certain limitations, such as slow convergence, which can be attributed to challenges like matching conflicts and limited supervision imposed by the matching process. This paper analyzes and identifies two forms of conflicts that arise from one-to-one matching: opposite optimization directions for similar samples and misalignment in query-object matching between different decoder layers. To mitigate the conflict while maintaining the end-to-end properties, we identify negative samples that closely resemble positive samples as shadow samples and ignore their classification loss during training. To address the issue of limited supervision, we compute the regression loss for these shadow samples, thereby providing additional localization supervision. By addressing these issues, our strategy enhances network training efficiency and improves overall performance under identical training configurations. Furthermore, we propose a loss-balancing strategy to enhance the effectiveness of shadow samples. Additionally, a feature-aware query initialization approach is proposed that offers the benefits of providing distinct features to shadow queries and strengthening the interaction between queries and image features. Experimental results demonstrate that our Shadow-DETR substantially boosts existing methods such as DAB-DETR, Deformable-DETR, and DINO while achieving comparable performance with SOTA methods.