ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study

Area of Science:

• Artificial Intelligence in Education
• Medical Education Technology
• Anatomy Learning Tools

Background:

• Large Language Models (LLMs) offer personalized learning but face challenges with accuracy and bias.
• Existing AI tools like ChatGPT may not meet the stringent accuracy requirements for specialized fields like anatomy.
• There is a need for reliable AI-driven educational resources in anatomy.

Purpose of the Study:

• To evaluate the accuracy and effectiveness of a custom AI chatbot (Anatbuddy) for anatomy education.
• To compare the performance of Anatbuddy against a general-purpose LLM (ChatGPT 3.5) in providing anatomy information.
• To assess the potential of Retrieval Augmented Generation (RAG) in creating specialized, accurate AI educational tools.

Main Methods:

• Developed Anatbuddy, an interactive AI chatbot using OpenAI API and Retrieval Augmented Generation (RAG).
• Programmed Anatbuddy with a curated knowledge base for context-aware responses.
• Administered 18 thoracic anatomy queries to both Anatbuddy and ChatGPT 3.5.
• Evaluated responses using a 5-point Likert scale by three experienced anatomists, with scores finalized by a blinded third party.

Main Results:

• Anatbuddy achieved significantly higher factual accuracy (mean 4.78/5) compared to ChatGPT 3.5 (mean 4.11/5).
• No significant differences were found between the chatbots in relevance, completeness, coherence, or fluency.
• The study highlights Anatbuddy's enhanced reliability for specific educational domains.

Conclusions:

• Custom AI chatbots, like Anatbuddy, are crucial for ensuring accuracy in specialized educational content.
• The anatomy profession should consider developing tailored AI solutions to overcome limitations of general LLMs.
• Further research should explore student adoption and learning impact of specialized anatomy chatbots.