Which curriculum components do medical students find most helpful for evaluating AI outputs?

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Large Language Models (LLMs) present both risks and opportunities in medical education due to their human-like communication.
• Future physicians require skills to critically assess AI-generated outputs for safe patient management.
• Investigating medical students' ability to evaluate LLM responses is crucial for adapting training.

Purpose of the Study:

• To assess medical students' proficiency in evaluating Large Language Model (LLM) responses to clinical scenarios.
• To identify prior learning experiences that aid students in scrutinizing AI-generated medical information.
• To gauge awareness of 'clinical prompt engineering' among future doctors.

Main Methods:

• A survey was administered to final-year medical students assessing the accuracy of GPT 3.5 responses to ten clinical vignettes.
• Content analysis was performed on the responses of 148 consenting medical students.
• Students identified training that enabled their evaluation of AI outputs.

Main Results:

• The median accuracy of students correctly evaluating LLM output was 56%.
• Interactive case-based and pathology teaching utilizing questions were reported as the most effective training.
• Familiarity with 'clinical prompt engineering' was very low, with only 5% of students aware of the concept.

Conclusions:

• Pathology and interactive case-based teaching are vital for equipping medical students to safely use LLM outputs.
• Medical training curricula should be updated to prepare graduates for AI-integrated healthcare environments.
• This study provides insights for designing effective medical education strategies in the age of AI.