Evaluating a Custom Chatbot in Undergraduate Medical Education: Randomised Crossover Mixed-Methods Evaluation of...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare Education

Background:

• Large language model (LLM) chatbots are increasingly used in medical education.
• Their pedagogical impact requires further evaluation.
• This study investigated a domain-specific chatbot's effect on medical students' performance, perception, and engagement.

Purpose of the Study:

• To assess the impact of a custom-built educational chatbot on medical students' academic performance.
• To evaluate students' perceptions of the chatbot compared to traditional study methods.
• To explore the chatbot's influence on cognitive engagement and learning.

Main Methods:

• A randomized crossover design involving 20 first-year medical students.
• Comparison between a custom chatbot (Lenny AI) and conventional study methods for academic tasks.
• Assessment via Single Best Answer (SBA) questions, post-task surveys, and focus groups.

Main Results:

• Students reported significantly higher satisfaction, engagement, and perceived quality with the chatbot (p < 0.05).
• Chatbot use correlated with increased perceived efficiency and confidence, but not with improved SBA scores.
• Qualitative analysis indicated faster factual recall but limited support for higher-order reasoning; students noted trust but desired transparency.

Conclusions:

• The educational chatbot enhanced usability and student satisfaction.
• No significant improvements in academic performance or higher-level cognitive skills were observed in the studied tasks.
• Future chatbot designs should incorporate adaptive scaffolding, transparent sourcing, and promote critical engagement for greater educational value.