Value of information in virtual patient performance evaluations

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Decision Analysis

Background:

• The American Board of Family Medicine (ABFM) incorporates virtual patients (VP) into its Maintenance of Certification program.
• Evaluating diagnostic performance in VPs may involve calculating the value of information (VOI).
• Primary care clinical scenarios often extend beyond simple diagnostic efficiency.

Purpose of the Study:

• To identify practical limitations in applying VOI scoring to virtual patient simulations in primary care.
• To explore the suitability of a Bayesian physician model for VOI assessment in these contexts.

Main Methods:

• Analysis of constraints within simulated primary care practice environments.
• Conceptual modeling of a Bayesian physician approach to VOI scoring.

Main Results:

• VOI scoring in primary care simulations is practically constrained by the complexity of user queries.
• A Bayesian physician framework requires explicit decision-making perspectives and extensive local data for effective VOI scoring.

Conclusions:

• Current VOI scoring methods may be insufficient for evaluating complex primary care simulations.
• The Bayesian physician model offers a theoretical framework but necessitates specific data and decision-making structures for practical application in VP assessments.