Reliable but Not Interchangeable: What Validated Instruments Measure When They Grade an Anastomosis

Area of Science:

• Microsurgery
• Surgical Skill Assessment
• Medical Education Technology

Background:

• Multiple validated microsurgical assessment instruments exist, differing in product-based (anastomosis quality) and process-based (technique execution) approaches.
• The interchangeability and comparative behavior of these instruments have not been directly examined.

Purpose of the Study:

• To compare the behavior of three validated microsurgical assessment instruments (ALI, MARS10, SMaRT) when applied to identical surgical performances.
• To evaluate interrater reliability, rank-order agreement, and score distributions across these instruments.

Main Methods:

• Forty-five participants performed a standardized vascular anastomosis on a chicken leg model.
• Two trained graders independently assessed performances using ALI, MARS10, and SMaRT.
• Interrater reliability (ICC), rank correlation (Spearman's rho, Kendall's W), and score effects were analyzed.

Main Results:

• All instruments demonstrated high interrater reliability (ICC > 0.939).
• Rank correlations varied, with strongest agreement between ALI and MARS10 (ρ=0.913).
• Instruments showed different score distributions, including ceiling effects (MARS10) and varied score variability (ALI).

Conclusions:

• Despite high reliability, microsurgical assessment instruments differ in rater variability, discriminatory capacity, and prioritized performance aspects, reflecting distinct definitions of surgical skill.
• Current assessment tools are not interchangeable, impacting training and research.
• Future technology-driven approaches may enhance objective and consistent surgical skill evaluation.