Virtual reality informs clinical observation tool.

Area of Science:

• Medical Education
• Health Professions Education
• Assessment in Medical Education

Background:

• Workplace-based assessments are crucial for trainee skill evaluation but often lack valid instruments.
• Previous expert-rater tools for assessing behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI) required specialized expertise.
• There is a need for psychometrically sound assessment tools usable by general clinicians.

Purpose of the Study:

• To establish validity evidence for a modified observational tool designed for clinicians to assess resident competencies in BHAG and MI.
• To adapt an existing expert-based instrument for broader clinical use.
• To explore the utility of virtual reality simulations in assessment tool development.

Main Methods:

• A previously developed expert instrument was modified through clinician group interviews.
• Recorded virtual reality (VR) simulations of resident-clinician interactions were used.
• Interrater reliability was assessed between clinicians using the modified tool.
• The tool's ability to discriminate between resident skill levels was evaluated using recorded VR simulations.

Main Results:

• Modifications included changing MI assessment from frequency counts to binary outcomes and adding an entrustment item.
• The clinician tool showed substantial to near-perfect agreement for BHAG items but variable agreement for MI items.
• The tool successfully discriminated between trained and untrained residents in BHAG skills (p=0.002) and entrustment levels (p=0.001).
• Differences in MI adherence were not statistically significant (p=0.095) with the clinician tool.

Conclusions:

• Virtual reality (VR) offers a promising method for generating validity evidence for assessment tools.
• The modified clinician tool demonstrates potential for workplace-based assessments of BHAG and MI skills.
• Further research is needed to evaluate the tool's performance in real-world clinical settings.