Evaluating artificial intelligence-generated multiple-choice questions in clinical pathology

Large language models are increasingly applied in medical education, but their role in clinical pathology remains uncertain. We conducted a prospective evaluation during the 2024-2025 academic year using GPT-4 with a standardized prompting protocol to generate multiple-choice questions for pathology resident education in clinical chemistry, hematopathology, clinical microbiology, molecular pathology, bioinformatics, and transfusion medicine. Faculty experts scored items for clarity and accuracy on a 1-5 scale and assigned a disposition of kept as is, kept but edited, or discarded. Of 260 items reviewed, 108 (41.5%) were retained without revision, 86 (33.1%) with edits, and 66 (25.4%) were discarded. Subspecialty performance varied, with molecular pathology showing the highest retention (60% unedited), while clinical microbiology had a relatively high discard rate (41.7%). Items retained without revision demonstrated significantly higher clarity and accuracy scores than discarded items (P < 0.05), and quality metrics also varied significantly across subspecialties (P < 0.001). Higher clarity (odds ratio: 1.68, 95% confidence interval: 1.12-2.54, P = 0.013) significantly predicted favorable disposition. Faculty comments revealed that AI-generated questions most often failed due to inappropriate difficulty calibration, lack of board relevance, and a disconnect from real-world laboratory practices. Reviewers favored practical, application-focused content over factual recall. Nearly three-quarters of artificial intelligence-generated questions were usable after expert review, highlighting the potential of large language models to accelerate assessment development. However, uneven subspecialty performance and risks of regulatory misalignment underscore the necessity of sustained expert oversight.