Consistency in grading clinical skills

Prelicensure nursing students have historically had differing experiences in a health assessment course and have identified inconsistencies among faculty grading physical examinations. The purpose of this study was to examine interrater agreement among nurse educators grading summative physical examinations performed by nursing students. Six nurse educators with experience in teaching and evaluating students in a health assessment course observed and independently graded a physical examination performed by a simulated student actor on a simulated patient actor. The physical examination was simultaneously audio-visually recorded. Approximately one month later, study participants re-graded the recording of the physical examination. Following this, interrater agreement was determined for live versus recorded grading. Acceptable levels of interrater agreement with both methods of grading were found. However, a discrepancy in pass/fail determinations among evaluators existed with both methods of grading. Measurement of percent agreement on individual checklist items revealed items with unacceptable interrater agreement, as well as several challenges with checklist documentation, giving direction for further evaluator training and checklist refinement in order to promote interrater agreement.