Comparing AI-Assisted Problem-Solving Ability With Internet Search Engine and e-Books in Medical Students With...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Educational Assessment

Background:

• Artificial intelligence (AI), especially large language models (LLMs), is increasingly impacting medical education.
• The effectiveness of AI-LLMs for students with diverse prior knowledge levels is not well understood.

Purpose of the Study:

• To evaluate the performance of medical students with and without formal pharmacology knowledge using AI-LLM GPTs, internet search engines, e-books, or self-knowledge for multiple-choice questions (MCQs).

Main Methods:

• A cross-sectional study involved 100 medical students (50 naive, 50 learned) at a tertiary care hospital.
• Participants answered MCQs using self-knowledge, e-books, Google, or ChatGPT-4o.
• Scores were compared using analysis of covariance, with self-knowledge as a covariate.

Main Results:

• Learned students outperformed naive students across all methods (P<.001), with the greatest effect size for AI-LLM GPTs.
• Performance hierarchy for all students: AI-LLM GPT > internet search engine > self-knowledge ≈ e-books.
• Naive students using AI scored higher than learned students using Google or e-books (P<.01).

Conclusions:

• AI-LLM GPTs enhance MCQ performance, especially for students with limited prior knowledge.
• AI tools can enable less knowledgeable students to outperform peers using traditional resources.
• Further research is needed on AI's impact on deep learning and critical thinking in medical education.