Development and Evaluation of an Artificial Intelligence-Powered Surgical Oral Examination Simulator: A Pilot Study

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Surgical Training

Background:

• Traditional surgical oral examinations face limitations in consistency and accessibility.
• There is a need for innovative tools to supplement faculty-led surgical training sessions.

Purpose of the Study:

• To develop and validate an AI-powered platform simulating surgical oral examinations.
• To assess the technical performance and educational utility of a novel large language model (LLM)-based tool for surgical education.

Main Methods:

• A cross-sectional study involving 12 surgical clerkship students.
• Technical validation of the surgery oral examination large language model (SOE-LLM) across 8 performance domains.
• Educational utility assessment using a 5-point Likert scale.

Main Results:

• The SOE-LLM demonstrated consistent performance as an oral examiner, accurately guiding students and providing clinically sound responses.
• The AI tool maintained examination fidelity, requiring diagnostic reasoning and differentiating management strategies.
• Students found the platform valuable for examination preparation (mean, 4.25) and practicing high-stakes decisions in a low-stakes environment (mean, 4.83).

Conclusions:

• The SOE-LLM shows significant potential as a valuable tool in surgical education.
• The platform offers a consistent, accessible, and effective method for simulating oral examinations.
• AI-driven simulation can enhance surgical training by providing safe practice environments.