Direct Observation vs. Video-Based Assessment in Flexible Cystoscopy

Area of Science:

• Medical Education
• Surgical Skills Assessment
• Urology

Background:

• Direct observation of clinical skills is limited by observer presence, time constraints, and potential bias.
• Video-based assessment offers a potential solution to overcome these limitations, enhancing flexibility and efficiency.

Purpose of the Study:

• To evaluate the reliability of video-based assessment for cystoscopy procedures.
• To compare potential bias in direct observation versus video-based assessment by blinded raters.

Main Methods:

• A blinded observational trial involving 20 medical students and 9 urologists performing cystoscopies.
• Recordings were rated by a direct observer and two blinded video-raters using a global rating scale (GRS).
• Intrarater and interrater reliability were assessed, and direct observer bias was analyzed using a paired samples t-test.

Main Results:

• High intrarater reliability (Pearson's r = 0.86) and interrater reliability (0.74 single measure, 0.85 average measures) were found for video-raters.
• A 'hawk-dove effect' was observed between the two video-raters.
• Significant direct observer bias was detected when comparing direct observation scores to blinded video-rater assessments (p < 0.001).

Conclusions:

• Video-based assessment, particularly with multiple raters, is a reliable method for evaluating cystoscopy skills.
• Direct observation is prone to significant bias compared to blinded video assessment, highlighting the advantages of video-based methods in medical education.