Conducting an objective structured clinical examination under COVID-restricted conditions

Area of Science:

• Medical Education
• Assessment Science
• Public Health

Background:

• The COVID-19 pandemic created significant challenges for administering performance assessments, particularly for medical certification and licensure exams.
• The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), underwent modifications to adapt to pandemic-related health restrictions.
• Gathering evidence for the reliability and validity of these modified assessments was crucial for maintaining the integrity of medical certification processes.

Purpose of the Study:

• To evaluate the reliability and validity of the modified NAC Examination administered during the COVID-19 pandemic.
• To provide evidence supporting the continued use and quality of the NAC Examination despite necessary adaptations.
• To analyze the impact of modifications on assessment outcomes across different demographic groups.

Main Methods:

• The modified NAC Examination was administered to 2,433 candidates in 2020-2021.
• Reliability was assessed using Cronbach's alpha, decision consistency, and accuracy.
• Validity evidence was gathered by comparing scores across demographic groups (gender, IMG status, PGT, language) and correlating NAC Examination scores with the Medical Council of Canada Qualifying Examination (MCCQE) Part I.

Main Results:

• Reliability estimates were comparable to previous administrations and similar OSCEs.
• Statistically significant differences in total and sub-scores were observed for gender, while IMG and PGT status showed significant sub-score differences.
• Language of examination did not significantly impact scores. Correlations between NAC Examination and MCCQE Part I scores supported validity arguments.

Conclusions:

• The modified NAC Examination demonstrated reliable total scores and pass/fail decisions.
• Score differences across demographic groups aligned with existing literature, supporting validity.
• Adaptations made to OSCEs for health restrictions can be implemented without compromising overall assessment quality.