[Evaluating the performance of generative AI in assisting the differential diagnosis of weight loss]
Ying Liu1, Yunhong Zhang2, Dongping Cai3
1Department of General Practice, the First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou 310003, China. 1511018@zju.edu.cn.
Objectives:
To systematically evaluate the performance of generative artificial intelligence (GenAI) models, DeepSeek-V3 and the Qwen3 series, in the differential diagnosis of weight loss.
Methods:
A search was conducted in the PubMed database for all case reports published in the American Journal of Case Reports between January 1, 2012 and June 2, 2025, containing the term "weight loss" in the title or abstract. Two senior general practitioners independently reviewed each case to determine whether it met predefined diagnostic criteria for weight loss (emaciation). Cases that did not meet these criteria, had incomplete information, or involved clearly defined specialty-specific diagnoses and treatments were excluded. The remaining cases were then compiled into standardized clinical case summaries. These summaries were presented to DeepSeek-V3 and the Qwen3 series models (Qwen3-235B-A22B, Qwen3-30B-A3B, and Qwen3-32B) to generate ranked lists of the top 10 differential diagnoses. The models were not specifically fine-tuned for this task. Sensitivity, precision, and F1-score were used to evaluate performance. Intergroup comparisons were performed using McNemar's test and Cochran's Q test.
Results:
A total of 87 case were analyzed. DeepSeek-V3 demonstrated better performance than Qwen3-235B-A22B in sensitivity, precision, and F1-score, especially at the Top5 level (P=0.043). Among the Qwen3 series models, Qwen3-235B-A22B showed the best performance in sensitivity, precision, and F1-score for the Top1 diagnosis, but the differences among the three Qwen3 models across all diagnostic levels were not statistically significant (all P>0.05).
Conclusions:
Domestic GenAI models exhibit a characteristic of "breadth over precision" in the differential diagnosis of weight loss, with DeepSeek-V3 performing better at key diagnostic levels. Although the sensitivity and precision for the top-ranked diagnosis require improvement, these models have the potential to serve as effective clinical decision support tools, broadening the diagnostic perspectives of general practitioners.
