Validity Evidence for STATUS to Assess Resident Tolerance for and Competence in Communicating Medical Ambiguity

No assessment instrument with validity evidence exists to assess resident competence in communicating medical ambiguity. Here, validity evidence was collected for STATUS (Scalable Tolerating Ambiguity/Uncertainty Tool Utilizing Simulation).

Using avatar patients and 2 simulated cases, investigators created a guidebook and trained 10 faculty in STATUS use. Pediatric residents completed 2 video-recorded simulated cases. Residents self-assessed tolerance for communicating medical ambiguity. Two faculty reviewed each video, assessing participants for communicating medical ambiguity. Validity evidence collected included the following: content, response process, internal structure, and relationship to other variables. Generalizability theory analysis was conducted to understand the assessment tools' reliability.

Of 89 eligible residents, 43 (48.3%) had sessions recorded. Eighty-six videos were analyzed. Faculty rater training increased inter-rater reliability by 0.34 units. The Φ-coefficient was 0.72 for the resident self-assessment tool and 0.26 for the faculty rater assessment tool. The decision study found that with 11 faculty raters and 11 scenarios, the Φ-coefficient would be 0.70. Resident self-assessment was negatively associated with faculty rater assessment with a Spearman correlation of -0.21 overall, indicating a possible weak correlation.

Results provide sufficient reliability to measure resident self-assessment of tolerance for communicating medical ambiguity. More scenarios would likely result in higher reliability for faculty assessment.