A Multi-AI Agent Framework for Interactive Neurosurgical Education and Evaluation: From Vignettes to Virtual...

Area of Science:

• Artificial Intelligence in Medicine
• Neurosurgical Education
• Clinical Reasoning Assessment

Background:

• Traditional medical board exams use static vignettes, unlike real-world clinical practice.
• Large language models (LLMs) show promise for interactive medical simulations.
• Neurosurgery presents unique challenges for LLMs due to variable patient communication and reliance on objective data.

Purpose of the Study:

• Develop and evaluate a multi-AI agent framework for neurosurgical case assessment.
• Create realistic clinical interactions using simulated patients and objective data access.
• Assess the diagnostic accuracy of GPT-4o in a novel conversational format.

Main Methods:

• A framework converted 608 neurosurgery questions into AI-driven conversations.
• Three specialized AI agents (patient, system, clinical) were utilized.
• GPT-4o's diagnostic accuracy was compared across vignettes, patient-only, and patient + system AI interactions, benchmarked against neurosurgery residents.

Main Results:

• GPT-4o performance significantly dropped in conversational formats compared to traditional vignettes (MC: 89.0% to 60.9%; free-response: 78.4% to 30.3%).
• Access to objective data via system AI improved GPT-4o performance (MC: 67.4%; free-response: 61.8%).
• Neurosurgery residents outperformed GPT-4o in free-response conversations (70.0% vs 28.3%) and found the interactive format highly educational.

Conclusions:

• The multi-AI agent framework challenges LLM evaluation and serves as an educational tool for neurosurgery.
• Conversational AI assessment may provide a more accurate measure of clinical reasoning than traditional multiple-choice tests.
• The interactive framework holds significant potential for advancing medical education and assessment in neurosurgery.