Performance evaluation of five large language models for assisting in the interpretation of urinalysis reports for...

Large language models (LLMs) show promise for interpreting laboratory reports, yet real-world validation remains limited. This study evaluated five advanced LLMs in interpreting urinalysis reports for kidney diseases using real-world clinical data, providing empirical evidence for the utility of LLM-assisted result interpretation.

We retrospectively collected 120 urinalysis reports from patients with primary glomerular diseases and secondary nephropathies. The testing platforms included the Sysmex UF5000 and Mindray EU8600. Five LLMs (ChatGPT-5, Claude-4.5, Gemini-2.5, DeepSeek-V3.1, Qwen-3) were tasked with interpreting reports across five functional dimensions. Four certified laboratory technologists and four licensed physicians evaluated outputs using a 5-point Likert scale across six quality dimensions. Statistical analyses employed Friedman and Wilcoxon signed-rank tests.

All five LLMs demonstrated clinical utility in interpreting urinalysis reports. Proprietary LLMs achieved higher overall scores (Claude-4.5: 4.78 ± 0.47; ChatGPT-5: 4.73 ± 0.50; Gemini-2.5: 4.69 ± 0.54) compared to open-source LLMs (DeepSeek-V3.1: 4.58 ± 0.66; Qwen-3: 4.57 ± 0.69). Across functional dimensions, the models performed proficiently in identifying abnormal parameters and analyzing their correlations, but suboptimally in interpreting instrument flags. Instrument-dependent variability was observed (Sysmex vs. Mindray, p<0.001). In quality assessments, Claude-4.5 exhibited the best overall performance, ChatGPT-5 excelled in accuracy and clarity, and Gemini-2.5 demonstrated strong practicality. Regarding safety, Claude-4.5 exhibited the lowest hallucination rate (7.5 %). Common hallucinations included misinterpretation, definition errors, and over-interpretation.

LLMs demonstrate significant capability in urinalysis interpretation, though proprietary models currently excel in reasoning and hallucination resistance. Instrument-specific flag interpretation and hallucination mitigation remain critical challenges requiring Retrieval-Augmented Generation (RAG) integration and human oversight.