Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service...

Large language models (LLMs) are increasingly evaluated using medical examination datasets, yet most studies emphasize overall accuracy rather than the psychometric structure of test items. We evaluated five LLMs on 199 text-only cardiology residency in-service examination items previously characterized using resident-derived psychometric metrics. Three frontier models were compared with two open-source comparators using a standardized zero-shot, repeated-query protocol and strict-majority scoring. Frontier models achieved substantially higher accuracy than open-source comparators, with Claude Opus 4.6, Gemini 3.1 Flash-Lite, and GPT-5.4 reaching 86.4%, 82.9%, and 81.9%, respectively, compared with 53.3% for MedQwen and 18.6% for Qwen-3.5-35B. Across frontier models, performance increased progressively from hard to easy resident-derived item strata. In multivariable analyses, item difficulty was the only classical psychometric factor consistently associated with AI correctness. IRT-based re-analysis confirmed that higher latent item difficulty was independently associated with lower frontier-model accuracy. Human-AI item-level correlations were modest but exceeded permutation-based null expectations, and frontier-model errors were concentrated among highly ranked human distractors. These findings show that item-level psychometric analysis helps explain variation in frontier LLM performance beyond overall accuracy alone. However, expert-rated rationale assessment and none-of-the-above perturbation testing revealed that strong examination accuracy did not guarantee high-quality explanatory support or reliable recognition of answer absence, indicating that examination performance and answer-selection robustness represent related but distinct dimensions of model behavior.