Automatic Final-Product Assessment of Virtual Reality Mastoidectomy Performance: A Validity and Reliability Study

Area of Science:

• Surgical Education
• Medical Simulation
• Otology

Background:

• Effective assessment is crucial for surgical training.
• Virtual reality (VR) temporal bone simulators offer automated metrics but lack comprehensive assessment of complex skills.
• Advanced automated assessment routines are needed for VR surgical simulation.

Purpose of the Study:

• To integrate and validate an automated final-product assessment for mastoidectomies in a VR simulator.
• To compare automated assessment with expert manual ratings for validity and reliability.

Main Methods:

• 33 trainees performed virtual mastoidectomies using the Visible Ear (VR) Simulator.
• A new automated rating routine based on the modified Welling Scale was implemented.
• Automated assessments were compared to expert manual ratings using agreement, intraclass correlation, and generalizability analysis.

Main Results:

• The automated system achieved 83.9% overall agreement with manual ratings, comparable to inter-rater reliability (88.9%).
• 15 out of 26 items showed high agreement (>85%) between automated and manual assessments.
• Generalizability analysis indicated five repetitions are needed for high-stakes assessment reliability (G coefficient >0.8).

Conclusions:

• An automated assessment system for VR temporal bone simulators is feasible, valid, and reliable.
• This system can serve as a valuable tool for self-directed surgical training and skills certification.