Medical student reliance on artificial intelligence in nephrology education

Large language models such as ChatGPT are increasingly used in medical education and may influence student learning and decision-making. Despite strong performance on factual recall, limitations in clinical reasoning raise concerns about how learners engage with artificial intelligence (AI)-generated recommendations, particularly in challenging domains such as renal physiology, where foundational understanding underpins clinical application.

Fifty-seven first-year medical students completed 24 multiple-choice questions during a pediatric nephrology and urology case-based learning session. Each clinical question was paired with a corresponding foundational science question and presented in random order. Students answered individually, reviewed a ChatGPT-generated answer (deliberately correct or incorrect), and then re-answered the question. Change-to-match behavior was recorded. Response-level analyses used generalized estimating equation logistic regression to account for repeated measures within students, with sensitivity analyses adjusting for item difficulty.

A total of 1357 paired responses were analyzed. Students changed their answers to match ChatGPT in 22.3% of cases. Change-to-match behavior was more frequent for foundational than clinical questions (24.1% vs. 20.4%). After adjusting for clustering and item difficulty, foundational question type remained associated with greater likelihood of change-to-match behavior (odds ratio [OR] 1.57, 95% confidence interval [CI] 1.08-2.29). Incorrect-to-correct answer changes exceeded correct-to-incorrect changes, resulting in a modest net improvement in accuracy.

Medical students selectively relied on AI-generated recommendations, particularly when uncertain and when engaging with foundational content. These findings suggest that reliance on AI is context-dependent and highlight the importance of domain-sensitive AI literacy in nephrology education.