大型语言模型在数值与语义医学知识中的表现:基于证据的问题和答案的横截面基准研究.
Eden Avnat1,2, Michal Levy3,4, Daniel Herstain1
1Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622.
Journal of medical Internet research
|July 14, 2025
概括
大型语言模型 (LLM) 在回答医疗问题方面表现不同,在语义任务方面表现出色,但与人类专家相比,在数值任务方面落后. 克劳德3和GPT-4展示了医学学科的不同优缺点.
科学领域:
- 人工智能在医学中的应用
- 自然语言处理自然语言处理.
- 临床决策支持 临床决策支持
背景情况:
- 临床问题解决依赖于语义和数值医学知识.
- 大型语言模型 (LLM) 在临床实践中显示出潜力,但在非语言基础任务中面临局限性.
- 代币化本质上可以限制LLM生成基于证据的答案的能力.
研究的目的:
- 评估LLM在数值和语义医学问题上的表现.
- 检查跨医学主题的LLM能力的模型内和模型间的差异.
- 将LLM的表现与人类医学专家进行比较.
主要方法:
- 开发了一个全面的医学知识图表,以创建基于证据的医学问题和答案 (EBMQAs).
- 基准GPT-4和Claude 3 Opus对24,000个EBMQA进行了基准测试,评估了数值和语义问题类型的准确性.
- 通过对LLM与人类医学专家进行100个数值问题的比较来验证绩效.
主要成果:
- 克劳德3和GPT-4在语义问题 (68.7%,68.4%) 的准确度高于数字问题 (61.3%,56.7%).
- 克劳德3在数值问题准确性 (P<.001) 方面表现优于GPT-4,在医疗子标签中表现有显著差异.
- 人类专家在准确度上明显超过了两种LLM (82.3%对64.3%和55.8%).
结论:
- 在医学问题上,LLM在语义问题上表现更好,而不是数值问题,Claude 3在数值准确性方面表现出优势.
- 这两种LLM在医学学科之间都存在绩效差距,低于人类专家.
- 律师事务所的回答或弃权决定不能可靠地预测准确性,需要谨慎对待他们的医疗建议.
相关概念视频
Improving Translational Accuracy
2.7K
2.7K
Classification of Illness
8.0K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
8.0K
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
102
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
102


