Reliability of the Clinical SBIRT Proficiency Checklist for Medical Students

The Substance Abuse and Mental Health Services Agency (SAMHSA) recognizes Screening, Brief Intervention, and Referral to Treatment (SBIRT) as a comprehensive approach to identify individuals with or at risk of developing substance use disorders. Few studies have explored tools for assessing medical student SBIRT competencies.

First-year medical students trained in SBIRT and Motivational Interviewing completed an Objective Structured Clinical Encounter (OSCE) with a standardized patient who presented with substance use. Six trained members of our research team reviewed 118 OSCE videos utilizing the Clinical SBIRT Proficiency Checklist (CSPC); additionally 30% (n = 37) were randomly selected for pair-review to examine interrater reliability.

Interrater reliability was Cohen's kappa of 0.89 for the presence of SBIRT skills and 0.39 agreement for the absence of skills. Across the videos, the most commonly observed skill was screening for alcohol use (75.4%, 95%CI 66.5, 84.3), while organizing referral for treatment was infrequently observed (36.4%, 95% CI 22.0, 50.8).

The CSPC is a reliable tool for assessing medical student SBIRT skills on an OSCE. These findings provide insights on medical student SBIRT knowledge and provides a practical tool for providing early clinically relevant feedback on these skills.