Uncertainty Quantification in Detection Transformers: Object-Level Calibration and Image-Level Reliability.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Detection Transformers (DETRs) offer end-to-end object detection but generate numerous predictions, challenging reliability assessment.
• Ensuring trustworthy predictions is crucial for safety-critical applications like autonomous driving.

Purpose of the Study:

• To investigate the reliability of predictions generated by DETRs.
• To introduce a novel metric for evaluating model calibration and post-processing effectiveness.
• To propose a framework for uncertainty quantification in DETR models.

Main Methods:

• Empirical and theoretical analysis of DETR prediction strategies.
• Investigation of the Hungarian matching algorithm's role in shaping DETR outputs.
• Development and validation of Object-level Calibration Error (OCE).
• Introduction of a post hoc uncertainty quantification (UQ) framework.

Main Results:

• DETRs employ a "specialist strategy" where one prediction per object is well-calibrated, while others suppress confidence.
• This strategy is an emergent property of the loss-minimizing Hungarian matching algorithm.
• Existing metrics like average precision and expected calibration error are insufficient for evaluating DETR reliability.
• Object-level Calibration Error (OCE) effectively evaluates calibration and identifies reliable predictions.
• A UQ framework is presented to predict per-image model accuracy.

Conclusions:

• DETR predictions have varying reliability levels due to an inherent "specialist strategy".
• Novel metric OCE and UQ framework are essential for reliable deployment of DETRs in safety-critical applications.
• Joint evaluation of model calibration and post-processing is necessary but requires new metrics.