Validation Evidence using Generalizability Theory for an Objective Structured Clinical Examination

Area of Science:

• Pharmacy Education
• Medical Education Assessment
• Psychometrics

Background:

• Performance-based assessments, such as Objective Structured Clinical Examinations (OSCEs), are crucial for evaluating student competency in pharmacy education.
• Rigorous validation of these assessments is essential, as educational decisions depend on their reliability and accuracy.
• Generalizability Theory (G-Theory) offers a robust framework for assessing the reliability of complex assessments like OSCEs, though its application in pharmacy education is limited.

Purpose of the Study:

• To demonstrate the application of G-Theory in evaluating the reliability of an OSCE used for assessing pharmacy students' practice-readiness.
• To identify sources of score variation and provide evidence for the validity of OSCE results.
• To explore methods for enhancing the reliability of OSCEs in pharmacy education.

Main Methods:

• A 14-station OSCE was administered to 97 third-year PharmD students over three weeks.
• Univariate and multivariate G-Theory analyses were conducted using G_Strings and mGENOVA software.
• The study examined score variance components and performed decision studies to determine optimal assessment designs for improved reliability.

Main Results:

• The overall g-coefficient (reliability) for the OSCE was 0.72, indicating mediocre reliability.
• G-Theory analysis identified that students contributed to score variation, but other factors were also significant.
• Decision studies revealed that increasing the number of stations per week to seven could achieve a g-coefficient greater than 0.80.

Conclusions:

• G-Theory provides vital evidence of reliability for validating OSCEs in pharmacy education.
• The current OSCE demonstrated mediocre reliability, highlighting the need for improvements, such as increasing the number of stations.
• Distributing stations across multiple weeks and revising problematic stations are practical strategies to enhance OSCE reliability.