Effect of large language model on diagnostic accuracy and clinical completeness among nephrology fellows managing...

Infections are the predominant etiologies of post-transplant mortality in India, yet a structured infectious disease curriculum for nephrology fellows is limited. We evaluated whether large language model (LLM)-augmented clinical reasoning could improve diagnostic accuracy, clinical completeness, and clinical efficiency in managing complex transplant infection scenarios.

This was a prospective, single-center, pre (LLM-unaided)-post (LLM-aided)-study (washout period of 7 days). We investigated 1200 responses generated on 30 expert-validated kidney transplant infectious disease vignettes (50 sub-questions). 12 final-year nephrology fellows participated voluntarily in the study. Intervention was Open AI ChatGPT-4o generated exemplar responses. Responses were scored for accuracy (0-100%), completeness (0-3 scale), and time-to-answer by blinded assessors. Inter-rater reliability and paired differences were also analyzed.

Median accuracy increased from 72% (65·5-78·5) in the unaided arm to 85·5% (79·0-91·0) in the aided arm with Δ =  + 13·0%, p-value < 0·001. Completeness improved from 1·9 (1·6-2·1) to 2·5 (2·3-2·8) (p-value < 0·001). Median time-to-answer decreased from 7.1 (5.9-8.4) to 5.9 (4.8-7.1) minutes (p = 0.002). The most significant gains occurred in fungal (+ 19·2%) and viral (+ 17·5%) infections, especially in immunosuppressant modulation and antifungal treatment protocols. Minimal or no benefit was observed in dual infection scenarios and niche dose-duration knowledge. Only a single clinically significant hallucination (3·3%) occurred.

Our findings primarily support the role of LLM-generated outputs as structured educational tools to enhance clinical reasoning training among nephrology fellows. Real-world clinical deployment requires additional validation, safety safeguards, and prospective outcome studies.