Comparing AI chatbot simulation and peer role-play for OSCE preparation: a pilot randomized controlled trial

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Clinical Skills Training

Background:

• Artificial intelligence (AI) applications in medical education are growing, yet their impact on interactive clinical competencies requires further investigation.
• This pilot study evaluates the feasibility and educational outcomes of AI chatbot simulations versus traditional peer role-play (PRP) for Objective Structured Clinical Examination (OSCE) preparation.
• It also shares practical insights from implementing a novel AI tool in a clinical skills training setting.

Purpose of the Study:

• To compare the effectiveness of AI chatbot-based simulations against traditional peer role-play (PRP) for preparing medical students for OSCEs.
• To assess the feasibility and educational impact of using AI chatbots in clinical skills training.
• To identify practical lessons learned from integrating AI tools into medical education.

Main Methods:

• Nineteen final-year Korean medicine students were randomized into an AI chatbot group (n=9) or a PRP group (n=10).
• Both groups participated in physical examination and clinical interview practice sessions tailored to their group.
• Performance on OSCE stations (dizziness, shoulder pain) was evaluated using a structured checklist, and learning experiences were assessed via questionnaires.

Main Results:

• While statistical significance was not reached, trends indicated PRP groups scored higher in history taking, whereas AI chatbot groups showed higher scores in patient education.
• Survey data revealed PRP participants valued interaction authenticity and the exam-like setting.
• AI chatbot users reported greater satisfaction with autonomy, repetitive practice opportunities, and structured feedback.

Conclusions:

• AI chatbot training and PRP exhibit complementary strengths for OSCE preparation.
• PRP is effective for performance-based skills and realistic communication practice.
• AI chatbots show promise for enhancing clinical reasoning through self-paced, reflective learning, suggesting a blended learning model may optimize clinical skills development.