The evaluation of clinical predictions. A method and initial application.

Area of Science:

• Medical Informatics
• Clinical Decision Making
• Rheumatology

Background:

• Clinical predictions are inherently probabilistic, not certain.
• Assessing physician predictive accuracy is crucial for evaluating healthcare technologies.

Purpose of the Study:

• To evaluate the skill of clinical rheumatologists in predicting patient outcomes using the accuracy coefficient.
• To compare physician prediction accuracy with current computer-based prediction methods.

Main Methods:

• Utilized the accuracy coefficient, a measure of probabilistic accuracy.
• Assessed the predictive performance of clinical rheumatologists based on assigned outcome probabilities.
• Correlated physician performance scores with their clinical experience levels.

Main Results:

• Physicians' accuracy scores demonstrated a positive correlation with their degree of clinical experience.
• The accuracy coefficient provided a sensitive assessment of the marginal benefit of diagnostic technologies.
• The best-performing physicians exceeded the accuracy of most tested computer prediction methods.

Conclusions:

• The accuracy coefficient offers a robust method for assessing physician predictive skills.
• This approach can evaluate the individualization of predictions to patient-specific characteristics.
• It has potential applications in physician performance assessment and technology evaluation.