ChatGPT versus UpToDate in Preclinical Medical Education: Cross-Sectional Analysis Using Term Frequency-Inverse...

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Preclinical Medical Training

Background:

• Generative AI tools, including ChatGPT, are increasingly adopted by medical students for self-directed learning.
• The reliability of these AI models as supplementary resources for preclinical education is not well-established.
• There is a lack of comparative studies between AI-generated content and established evidence-based medical references like UpToDate.

Purpose of the Study:

• To evaluate the similarity between responses generated by ChatGPT (GPT-4o mini) and UpToDate for preclinical medical education questions.
• To assess the potential of ChatGPT as an adjunctive learning tool in preclinical medical education.

Main Methods:

• A cross-sectional comparison study was conducted using 150 preclinical medical questions.
• ChatGPT responses were generated across 10 separate sessions per question.
• Responses were preprocessed and compared with UpToDate content using TF-IDF cosine similarity, with comparisons against randomized text to assess nonrandom overlap.

Main Results:

• ChatGPT responses showed statistically significant similarity to UpToDate in 59.3% of questions.
• Pharmacology exhibited the highest concordance (mean cosine similarity 0.338), followed by pathology, biochemistry, microbiology, and immunology.
• All subject-level similarity scores surpassed those from randomized text, indicating nonrandom overlap.

Conclusions:

• ChatGPT (GPT-4o mini) demonstrates moderate, meaningful alignment with UpToDate for preclinical topics, excelling in fact-based subjects like pharmacology.
• While not a substitute for evidence-based resources, ChatGPT can function as an accessible supplementary learning tool for medical students.
• Responsible integration of AI in preclinical learning necessitates AI literacy training for critical appraisal and effective use.