ECOSBot: a multicenter validation pilot study of a generative AI tool for OSCE-based nephrology training

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Nephrology Training

Background:

• Nephrology diagnostic reasoning is complex, challenging traditional training methods.
• Objective Structured Clinical Examinations (OSCEs) are crucial but resource-intensive.
• Generative AI's potential for nephrology OSCE simulation is unexplored.

Purpose of the Study:

• To develop and assess ECOSBot, an AI tool for simulating nephrology OSCEs.
• To evaluate ECOSBot's performance in patient simulation and examiner roles.
• To determine the usability and utility of ECOSBot for medical students.

Main Methods:

• Developed ECOSBot using GPT-4o for standardized patients and examiners.
• Conducted a multicenter prospective study with 91 undergraduate medical students.
• Evaluated ECOSBot against faculty ratings for authenticity, correctness, relevance, and examiner scoring.

Main Results:

• ECOSBot achieved high fidelity in patient simulation (authenticity 98.6%, correctness 98.3%, relevance 99.2%).
• Strong agreement between ECOSBot and human raters for global scores (ICC=0.94).
• Students found ECOSBot highly useful (91.7%) for nephrology OSCE preparation.

Conclusions:

• ECOSBot reliably simulates nephrology OSCE roles with high fidelity.
• The tool aligns well with expert ratings, offering a scalable solution.
• Challenges remain in assessing subjective skills, but ECOSBot enhances nephrology education.