Methods to measure clinicians' diagnostic calibration: an appraisal for real world application

Many, if not most, diagnostic errors go unrecognized by clinicians unless they cause significant patient harm. Reducing the gap between clinicians' perceived and actual diagnostic performance (i.e., improving calibration) may improve patient outcomes. However, existing approaches to measuring calibration were developed for use in experimental settings rather than realistic practice environments. We reviewed the scientific literature and solicited expert field input to improve approaches for assessing calibration in clinical practice.

We performed a targeted literature review to identify studies that compared clinicians' diagnostic performance or accuracy to perceived accuracy or confidence. Key findings from the review were provided to 10 subject matter experts in clinical reasoning and diagnostic excellence. We then interviewed these subject matter experts in a series of four small-group teleconferences. Transcripts of group discussions were analyzed for thematic content relevant to improving the measurement of calibration in real-world care settings.

Discussions with experts highlighted that, while calibration reflects skill and expertise relevant to the task at hand, it is also fluctuating and context-dependent. Experts recommended that measurement of calibration should focus on clinically meaningful decisions that may be upstream of a final, correct diagnosis. Experts also recommended assessment of contextual factors that shape diagnostic decision-making and self-assessment.

Measures of calibration should focus not only on outcomes but also on processes, which are often more actionable and proximal to clinical decision-making. In response to expert input, we propose a measure of real-world calibration that builds upon a previous measure to include several contextual variables.