Remote and onsite scoring of OSCEs using generalisability theory: A three-year cohort study

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Traditional onsite scoring in Objective Structured Clinical Examinations (OSCEs) may be influenced by audience effects, potentially impacting student performance.
• Exploring alternative scoring methods like remote scoring (RS) is crucial for enhancing the reliability and feasibility of high-stakes medical assessments.

Purpose of the Study:

• To analyze the reliability and agreement between onsite scoring (OS) and remote scoring (RS) in a national high-stakes OSCE.
• To investigate factors influencing scoring accuracy in remote settings and identify potential barriers or facilitators.

Main Methods:

• Utilized generalizability theory to analyze scoring reliability from data collected from 154 students and 84 raters over three years (2013-2015).
• Randomly selected six stations from a 12-station OSCE and investigated perceptions affecting remote scoring.
• Employed Cronbach's alpha and kappa statistics to assess internal consistency and inter-rater agreement.

Main Results:

• Demonstrated moderate agreement (kappa = 0.623) and good reliability (G value = 0.93) between onsite and remote scoring.
• Identified students, locations, and raters as sources of variance in scoring.
• Technical feasibility, facilitation of wellbeing, and observational/attention deficits explained 73.886% of the variance in remote scoring.

Conclusions:

• Remote scoring offers a viable alternative to onsite scoring in OSCEs, providing good reliability and moderate agreement.
• Validated key factors for successful remote scoring, including operational feasibility and rater quality.
• Remote scoring effectively overcomes geographical limitations and mitigates the audience effect, enhancing assessment flexibility.