作为大众的医疗助理的LLM的可靠性:一项随机预先注册的随机研究
Andrew M Bean1, Rebecca Elizabeth Payne2,3,4, Guy Parsons1,5
1Oxford Internet Institute, University of Oxford, Oxford, UK.
Nature medicine
|February 9, 2026
概括
大型语言模型 (LLM) 在医疗保健中表现有希望,但在公众使用医疗建议时失败. 用户互动挑战意味着LLM在现实医学场景中表现不如对照组.
科学领域:
- 人工智能在医学中的应用
- 人与计算机的交互
- 临床决策支持 临床决策支持
背景情况:
- 医疗保健提供者正在调查公共医疗咨询的大型语言模型 (LLM).
- 法学士在医学考试中表现出色,但现实世界的表现尚未得到证实.
- 用户互动动态可能会影响医疗保健环境中的LLM有效性.
研究的目的:
- 评估LLM在协助公众进行医学诊断和处置方面的有效性.
- 在模拟的医疗场景中,将LLM协助与对照组进行比较.
- 确定在公共医疗咨询中部署LLM的挑战.
主要方法:
- 一项受控研究涉及1298名参与者,评估了10个医疗场景.
- 参与者得到了LLM (GPT-4o,Llama 3,Command R+) 或他们自己的资源 (控制) 的帮助.
- 性能通过条件识别和处置选择的准确性来衡量.
主要成果:
- 只有LLM能够准确地识别疾病 (94.9%) 和情绪 (56.3%).
- 使用LLM的参与者在识别疾病 (<34.5%) 或情绪 (<44.2%) 方面没有比对照组有任何改善.
- 与独立的LLM功能相比,用户交互显著降低了LLM性能.
结论:
- 在模拟医疗场景中LLM的表现并不能转化为有效的公共援助,因为用户交互挑战.
- 目前的基准没有预测与人类用户的LLMs的现实世界失败.
- 在部署LLM用于公共卫生保健咨询之前,系统的人类用户测试至关重要.
相关概念视频
01:29Reliability and Validity
14.1K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.1K
01:20Bioequivalence Experimental Study Designs: Completely Randomized and Randomized Block Designs
251
Body:Bioequivalence experimental study designs are crucial methodologies used in evaluating and comparing the bioavailability of different drug products. These designs are categorized into various types: completely randomized, randomized block, repeated measures, cross and carry-over, and Latin square designs.Completely randomized designs involve randomly allocating treatments to all subjects participating in the experiment. This allocation is achieved by assigning unique random numbers to...
251
01:04Random Error
9.8K
Random or indeterminate errors originate from various uncontrollable variables, such as variations in environmental conditions, instrument imperfections, or the inherent variability of the phenomena being measured. Usually, these errors cannot be predicted, estimated, or characterized because their direction and magnitude often vary in magnitude and direction even during consecutive measurements. As a result, they are difficult to eliminate. However, the aggregate effect of these errors can be...
9.8K
01:09Random Variables
17.9K
A random variable is a single numerical value that indicates the outcome of a procedure. The concept of random variables is fundamental to the probability theory and was introduced by a Russian mathematician, Pafnuty Chebyshev, in the mid-nineteenth century.
Uppercase letters such as X or Y denote a random variable. Lowercase letters like x or y denote the value of a random variable. If X is a random variable, then X is written in words, and x is given as a number.
For example, let X = the...
Uppercase letters such as X or Y denote a random variable. Lowercase letters like x or y denote the value of a random variable. If X is a random variable, then X is written in words, and x is given as a number.
For example, let X = the...
17.9K
01:13Randomized Experiments
9.1K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
9.1K
01:23Inhaled Medications
818
Inhaled medications are crucial for managing chronic obstructive pulmonary disease (COPD) and asthma. They are essential for effective treatment and control, ensuring optimal respiratory health and well-being. Inhaled medication delivers drugs directly to the lungs, providing a rapid onset of action and reducing systemic side effects compared to oral or injectable medications. Three primary types of inhalation devices are used to administer these medications: nebulizers, metered-dose inhalers...
818