Estimating diagnostic accuracy under uncertainty about disease status: a sepsis case study

Area of Science:

• Medical diagnostics
• Clinical decision-making
• Biostatistics

Background:

• Expert panels traditionally use dichotomous classifications (present/absent) for target conditions.
• This method can introduce bias when experts are uncertain.
• Probabilistic estimates offer a potential solution to mitigate bias.

Purpose of the Study:

• To compare diagnostic accuracy estimation methods using probabilistic expert judgments.
• To evaluate the impact of expert uncertainty on accuracy estimates.
• To assess the performance of different approaches in a real-world case study.

Main Methods:

• Compared three methods: dichotomous, direct weighting, and Bayesian approaches.
• Utilized probabilistic target condition presence estimates from expert panels.
• Applied methods to the SPACE study data on sepsis prediction models (SIRS, qSOFA, MEWS) and clinical bedside judgment (CBJ).

Main Results:

• Expert panels expressed significant uncertainty in 65% of cases (probability between 0.2-0.8).
• The Bayesian approach yielded different diagnostic accuracy estimates compared to the dichotomous approach.
• For SIRS, sensitivity/specificity estimates varied from 95%/46% (dichotomous) to 99%/60% (Bayesian).

Conclusions:

• Probabilistic estimates from expert panels reveal uncertainty often missed by dichotomous classifications.
• The Bayesian approach provides valid diagnostic accuracy estimates that account for expert uncertainty.
• Expert probabilities should accurately reflect true probabilities for reliable results.