Reliable Assessment of Healthcare Procedures: A Comparison of On-Site and Video-Based Methods

Assessing performance is critical in simulation training programs. Traditionally, predefined evaluation forms completed by experts are used, but this approach may introduce bias. Common strategies to mitigate bias include averaging scores from multiple evaluators or using video recordings to minimize disagreements among assessors. This study compares performance scores obtained through real-time on-site observation, independent video assessment, and collaborative video assessment of venipuncture performance. Additionally, we evaluate whether combining these methods enhances scoring consistency.

Eighteen medical students were invited to perform venipuncture trials, which were evaluated in three stages. Two evaluators observed and scored each trial on-site using a predefined evaluation form. The trials were video-recorded. At 12 weeks post-training, the evaluators independently reviewed the videos and assigned performance scores. At 14 weeks, they collaboratively reviewed the videos and provided a joint performance score. The Intraclass Correlation Coefficient (ICC) was used to assess the consistency between evaluators.

Mean scores differed significantly among the three assessment methods (P = 0.037), with independent video assessment yielding higher scores (75.3 ± 5.2) than on-site assessment (66.7 ± 6.1). Inter-rater reliability (ICC) ranged from 0.706 to 0.883, but was not statistically compared across methods.While collaborative assessment showed the highest consistency (0.889), implementing a multimodal approach would substantially increase faculty workload, as it requires scoring each student multiple times.

Video-based assessments, particularly collaborative review, enable detailed and repeated analysis of procedural skills, improving scoring consistency. However, the feasibility and workload implications of combining multiple methods must be considered before implementation in training programs.