Inter-Rater Reliability in a Pre-Graduation Nursing Objective Structured Clinical Examination: A Kappa and...

Objective structured clinical examinations (OSCEs) are widely used to assess nursing students' clinical skills. However, ensuring consistent scoring between examiners remains difficult, particularly for subjective areas such as communication.

This study aimed to evaluate inter-rater agreement between two faculty examiners in a pre-graduation OSCE recommended for final-year pre-registration nursing students in Japan, comparing technical skill stations with communication-focused stations.

A total of 90 final-year nursing students completed two OSCE stations: one assessing technical procedures, the other assessing communication and patient education. Two examiners independently scored each aspect of performance using binary checklists. Inter-rater agreement was calculated using Cohen's kappa and prevalence- and bias-adjusted kappa.

Higher inter-rater agreement was found for psychomotor items (e.g., auscultation) than for verbal or empathy-based items. In the technical station, agreement improved across successive circuits, suggesting examiner calibration. In contrast, in the communication station, agreement remained consistently low. Empathy-related items showed the greatest discrepancy between kappa and prevalence- and bias-adjusted kappa, highlighting challenges in evaluating subjective skills.

OSCE inter-rater reliability was higher for objective technical skills than for subjective communication skills and empathy-related behaviors, among pre-registration nursing students.

Improving checklist clarity and providing targeted examiner training for communication and empathy-related items may enhance the reliability of OSCE scoring in nursing education.