Related Experiment Video
Updated: Jul 17, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Exploring the Potential and Limitations of Chat Generative Pre-trained Transformer (ChatGPT) in Generating
Ibraheim Ayub1, Dathan Hamann2, Carsten R Hamann3,4
1Dermatology, A.T. Still University School of Osteopathic Medicine, Mesa, USA.
ChatGPT has limitations as a dermatology study tool. A study found only 40% of AI-generated board exam questions were accurate, highlighting the need for expert oversight in medical education.
Area of Science:
- Medical Education
- Artificial Intelligence in Medicine
- Dermatology
Background:
- Continuing medical education is vital for dermatologists.
- Artificial intelligence (AI) language models like ChatGPT offer potential tools for learning.
- Evaluating the efficacy of AI in generating specialized medical exam questions is crucial.
Purpose of the Study:
- To assess the limitations of ChatGPT as a study tool for dermatology board examinations.
- To evaluate the accuracy, complexity, and clarity of AI-generated questions based on dermatology literature.
Main Methods:
- Utilized ChatPDF to generate American Board of Dermatology Applied Exam (ABD-AE)-style questions from Journal of the American Board of Dermatology articles.
- Conducted a qualitative analysis of 40 generated questions by two board-certified dermatologists.
- Assessed questions for accuracy, complexity, clarity, and appropriateness for board exam preparation.
Main Results:
- Only 16 out of 40 (40%) AI-generated questions were deemed accurate and suitable for ABD-AE study.
- Significant limitations identified included low complexity, lack of clarity, and factual inaccuracies.
- ChatGPT struggled with domain-specific knowledge, contextual understanding, and generating effective distractors.
Conclusions:
- ChatGPT's current capabilities are insufficient for creating high-quality dermatology board-style questions.
- Expert dermatologists and medical educators remain essential for developing effective assessment tools.
- AI may assist in generating basic questions but cannot replace specialized medical expertise.
More Related Videos
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
04:09Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma
Published on: October 10, 2018
