Validity Evidence for STATUS to Assess Resident Tolerance for and Competence in Communicating Medical Ambiguity

Area of Science:

• Medical Education
• Healthcare Communication
• Simulation-Based Assessment

Background:

• Effective communication of medical ambiguity is crucial for patient safety and resident training.
• Existing assessment tools lack validity evidence for evaluating this specific competence.

Purpose of the Study:

• To collect validity evidence for the Scalable Tolerating Ambiguity/Uncertainty Tool Utilizing Simulation (STATUS).
• To assess resident competence in communicating medical ambiguity using a novel simulation-based tool.

Main Methods:

• Developed and implemented the STATUS tool with avatar patients and simulated cases.
• Trained faculty raters and collected data from pediatric residents' self-assessments and faculty evaluations.
• Analyzed validity evidence including content, response process, internal structure, and relationships with other variables.
• Utilized generalizability theory to evaluate assessment reliability.

Main Results:

• The resident self-assessment component of STATUS demonstrated a Φ-coefficient of 0.72.
• Faculty rater assessment yielded a Φ-coefficient of 0.26, with potential improvement to 0.70 using 11 raters and 11 scenarios.
• Faculty rater training improved inter-rater reliability by 0.34 units.

Conclusions:

• The STATUS tool provides sufficient reliability for measuring resident self-assessment of tolerance for communicating medical ambiguity.
• Additional simulated scenarios are recommended to enhance the reliability of faculty-based assessments.