语义临床人工智能与本地大语言模型表现在USMLE上的表现
Peter L Elkin1,2, Guresh Mehta1, Frank LeHouillier1,2
1Department of Biomedical Informatics, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, New York.
JAMA network open
|April 22, 2025
概括
语义临床人工智能 (SCAI) 与检索增强生成 (RAG) 显著改善了美国医学执照考试 (USMLE) 问题的大型语言模型 (LLM) 性能. SCAI RAG提高了准确性,有助于在医疗保健中实施LLM.
科学领域:
- 人工智能在医学中的应用
- 自然语言处理自然语言处理.
- 医疗教育 技术 技术 医学教育
背景情况:
- 大型语言模型 (LLM) 在医疗保健中越来越多地使用,但需要提高准确性和方法来随着时间的推移保持性能.
- 在标准化医学考试上评估LLM的表现对于评估他们的临床准备至关重要.
研究的目的:
- 确定是否纳入正式表示的语义临床知识可以提高美国医学执照考试 (USMLE) 的LLM绩效.
主要方法:
- 一项比较有效性研究评估了三个Llama LLMs (13B,70B,405B) 具有和没有语义临床人工智能 (SCAI) 检索增强生成 (RAG) 的比较有效性研究.
- 通过使用USMLE第一,第二和第三步的基于文本的问题来评估LLM绩效,并根据官方答案密钥确定准确性.
主要成果:
- SCAI RAG显著提高了USMLE第一,第二和第三步的LLM绩效.
- 13B LLM 通过 SCAI RAG (60.2%) 达到第三步的合格门.
- 70B和405B法学士通过了所有步骤,无论是否有SCAI RAG,70B模型在第一步达到92.0%,405B模型在第三步达到95.1%.
结论:
- 通过SCAI RAG进行语义临床知识整合,提高了医学执照考试的LLM准确性.
- 增加有针对性的临床知识的LLM显示出改善医疗问答能力和促进医疗保健实施的希望.
- 语义推理是提高关键医疗应用中的LLM性能的一个关键进步.
更多相关视频
05:56Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application
Published on: April 14, 2023
2.3K
09:53Quantifying the Brain Metastatic Tumor Micro-Environment using an Organ-On-A Chip 3D Model, Machine Learning, and Confocal Tomography
Published on: August 16, 2020
6.8K
相关概念视频
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
26
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
26
02:16Stereotype Content Model
13.9K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
13.9K