A Training Program Using an Audience Response System to Calibrate Dental Faculty Members Assessing Student Clinical
Michael J Metz1, Cynthia J Metz2, Marcelo T Durski2
1Dr. Michael Metz is Vice Chairman, Department of General Dentistry and Oral Medicine, University of Louisville School of Dentistry; Dr. Cynthia Metz is Assistant Professor, Department of Physiology and Biophysics, University of Louisville School of Medicine; Dr. Durski is Assistant Professor, Department of General Dentistry and Oral Medicine, University of Louisville School of Dentistry; Mr. Aiken is a Doctorate in Medical Dentistry (DMD) student and Master in Oral Biology candidate, University of Louisville School of Dentistry; Dr. Mayfield is Associate Dean of Clinical Affairs, Department of General Dentistry and Oral Medicine, University of Louisville School of Dentistry; and Dr. Lin is Associate Professor, Department of Oral Health and Rehabilitation, University of Louisville School of Dentistry. michael.metz@louisville.edu.
Abstract:
The aim of this study was to assess the effectiveness of calibration training of departmental faculty and competency graders using an audience response system on operative dentistry concepts across 12 months. The training sessions were designed to further solidify the process and equilibration of clinical opinions among faculty members and provide a more calibrated grading assessment during patient care for student performance feedback. Four (quarterly) calibration sessions occurred over 12 months in 2015. The first session was considered the baseline (control value) for this study. Pre- and post-calibration interrater agreement was assessed. Additionally, a pre and post assessment with ten Likert-scale questions was used to measure students' perceptions of instructional consistency. The results showed that a statistically significant increase in conceptual knowledge scores occurred for both departmental faculty members and competency graders across each of the four sessions (one-factor ANOVA; p<0.05). Interrater reliability agreement also significantly improved for both department faculty members and competency graders' clinical assessments over 12 months of implementation (Cohen's Kappa; p<0.05). There was a statistically significant increase in positive student perceptions on all ten questions (dependent t-test; p<0.05). Implementation of an audience response system for departmental and competency graders was found to be effective in facilitating a discussion forum, calibrating clinical assessments, and improving student perceptions. The positive results from this study support the value of dental schools' introducing faculty development programs to ensure consistent instruction for assessing dental student competence.
