MedAlign:一个临床医生生成的数据集,用于使用电子医疗记录进行指导跟踪
Scott L Fleming1,2, Alejandro Lozano1, William J Haberkorn3,4
1Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA.
Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence
|January 26, 2026
概括
在医疗保健中评估大型语言模型 (LLM) 很困难. 新的MedAlign数据集揭示了LLM生成电子健康记录数据的高错误率,突出了需要更好的评估方法的需要.
科学领域:
- 人工智能在医学中的应用
- 医疗保健的自然语言处理.
- 临床信息学 临床信息学
背景情况:
- 大型语言模型 (LLM) 显示出降低医疗保健行政负担和提高医疗保健质量的前景.
- 在现实的医疗保健文本生成任务上评估LLM,特别是使用电子健康记录 (EHR) 数据,由于现有数据集的局限性而具有挑战性.
- 目前的EHR问答数据集并不完全代表临床医生的复杂信息需求或文档负担.
研究的目的:
- 推出MedAlign,一个新的基准数据集,旨在评估LLM对EHR数据的评估.
- 通过使用MedAlign数据集,评估一般领域的LLM在实际临床指令后的任务上的表现.
- 探索人类临床医生排名和LLM评估的自动化指标之间的相关性.
主要方法:
- 开发了MedAlign,数据集包括983个自然语言指令用于EHR数据,由7个专业的15名临床医生策划.
- 包括临床医生为303个指令和276个纵向EHR的书面参考答案.
- 使用MedAlign评估了6个通用领域的LLM,临床医生对生成的响应的准确性和质量进行了排名.
主要成果:
- 在LLM中,错误率很高,从GPT-4的35%到MPT-7B-Instruct的68%不等.
- 在将上下文长度从32k缩短到2k令牌时,GPT-4的准确性下降了8.3%.
- 在临床医生排名和自动化自然语言生成指标之间发现了相关性,这表明自动化LLM评估的潜力.
结论:
- 现有的通用LLM在基于临床指令准确处理和生成EHR数据方面存在重大局限性.
- MedAlign数据集为开发和评估针对医疗保健中的特定临床医生的需求和偏好量身定制的LLM提供了至关重要的资源.
- 需要进一步的研究来提高LLM的性能,并为临床应用开发可靠的自动化评估方法.
相关概念视频
01:28Electron Transport Chains
112.0K
The final stage of cellular respiration is oxidative phosphorylation that consists of two steps: the electron transport chain and chemiosmosis. The electron transport chain is a set of proteins found in the inner mitochondrial membrane in eukaryotic cells. Its primary function is to establish a proton gradient that can be used during chemiosmosis to produce ATP and generate electron carriers, such as NAD+ and FAD, that are used in glycolysis and the citric acid cycle.
The ETC is comprised of...
The ETC is comprised of...
112.0K
01:24Electron Carriers
91.7K
Electron carriers can be thought of as electron shuttles. These compounds can easily accept electrons (i.e., be reduced) or lose them (i.e., be oxidized). They play an essential role in energy production because cellular respiration is contingent on the flow of electrons.
Over the many stages of cellular respiration, glucose breaks down into carbon dioxide and water. Electron carriers pick up electrons lost by glucose in these reactions, temporarily storing and releasing them into the electron...
Over the many stages of cellular respiration, glucose breaks down into carbon dioxide and water. Electron carriers pick up electrons lost by glucose in these reactions, temporarily storing and releasing them into the electron...
91.7K
03:07Electron Affinity
43.2K
The electron affinity (EA) is the energy change for adding an electron to a gaseous atom to form an anion (negative ion).
43.2K
01:23Inhaled Medications
785
Inhaled medications are crucial for managing chronic obstructive pulmonary disease (COPD) and asthma. They are essential for effective treatment and control, ensuring optimal respiratory health and well-being. Inhaled medication delivers drugs directly to the lungs, providing a rapid onset of action and reducing systemic side effects compared to oral or injectable medications. Three primary types of inhalation devices are used to administer these medications: nebulizers, metered-dose inhalers...
785
00:54Electron Behavior
108.0K
Overview
Electrons are negatively charged subatomic particles that are attracted to an orbit around the positively-charged nucleus of an atom. They reside in locations that are associated with energy levels called shells and are further organized into sub-shells and orbitals within each shell.
Electrons Orbit the Nucleus
Electrons are found in specific locations outside of the nucleus. The shell in which an electron resides indicates the general energy level of the electron: those closer to the...
Electrons are negatively charged subatomic particles that are attracted to an orbit around the positively-charged nucleus of an atom. They reside in locations that are associated with energy levels called shells and are further organized into sub-shells and orbitals within each shell.
Electrons Orbit the Nucleus
Electrons are found in specific locations outside of the nucleus. The shell in which an electron resides indicates the general energy level of the electron: those closer to the...
108.0K
01:18Electron Orbital Model
72.0K
Orbitals are the areas outside of the atomic nucleus where electrons are most likely to reside. They are characterized by different energy levels, shapes, and three-dimensional orientations. The location of electrons is described most generally by a shell or principal energy level, then by a subshell within each shell, and finally, by individual orbitals found within the subshells.
The first shell is closest to the nucleus, and it has only one subshell with a single spherical orbital called the...
The first shell is closest to the nucleus, and it has only one subshell with a single spherical orbital called the...
72.0K