临床诊断中大型语言模型的比较分析:跨常见和复杂医疗病例的性能评估
Mehmed T Dinc1, Ali E Bardak2, Furkan Bahar3
1Department of Medicine, Boston Medical Center, Boston, MA 02118, United States.
JAMIA open
|June 13, 2025
概括
领先的大型语言模型 (LLM) 在常见和复杂的临床病例中表现出高的诊断准确性,特别是Claude 3.7. 需要进一步的研究来将这些AI工具整合到临床实践中,以改善患者护理.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 临床决策支持 临床决策支持
背景情况:
- 大型语言模型 (LLM) 显示出临床决策支持的前景.
- 在现实场景中评估LLM诊断准确度至关重要.
- 现有的研究往往缺乏对不同模型和案例复杂性的系统比较.
研究的目的:
- 系统地评估和比较领先的LLMs的诊断性能.
- 评估LLM在提高临床推理和诊断准确性的潜力.
- 在常见和复杂的临床场景中分析表现.
主要方法:
- 评估了高级LLM (克劳德,GPT,Gemini) 的诊断能力.
- 利用60个常见和104个复杂的临床病例,分阶段披露信息.
- 在每个披露阶段评估了初级和差异诊断的准确性.
主要成果:
- 在常见情况下,LLM的准确性达到90%以上;在某些情况下,Claude 3.7的准确性达到100%.
- 克劳德3.7领导复杂的案件,在最后阶段准确率为83.3%.
- 在常见的场景中,较小的模型的性能与较大的模型相比较.
结论:
- 领先的LLM在各种临床环境中表现出显著的诊断准确性.
- 在这次评估中,克劳德3.7成为了表现最好的人.
- 翻译性研究和实施框架对于将AI整合到临床实践和医学教育中至关重要.
相关概念视频
Comparing the Survival Analysis of Two or More Groups
162
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...
162
Receiver Operating Characteristic Plot
104
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
104
Sensitivity, Specificity, and Predicted Value
225
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
225
Methods of Documentation VI: Case Management Model
564
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
564
Language and Cognition
338
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
338


