将5个生成性AI聊天机器人对LLM生成的临床问题的答案与医学信息科学家的证据摘要进行比较
Mallory N Blasingame1, Taneya Y Koonce1, Annette M Williams1
1Center for Knowledge Management, Vanderbilt University Medical Center, Nashville, TN, United States.
medRxiv : the preprint server for health sciences
|October 3, 2025
概括
五个大型语言模型 (LLM) 聊天机器人显示在临床问题上与医学信息科学家相似. 虽然两者都提供了额外的细节,但科学家们包括了更多,需要未来的有效性评估.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 临床决策支持 临床决策支持
背景情况:
- 大型语言模型 (LLM) 越来越多地用于临床问题解答.
- 对人类专家进行LLM绩效评估对于安全融入医疗保健至关重要.
研究的目的:
- 为了比较五个领先的LLM聊天机器人对临床问题的答案的准确性和完整性,与医学信息科学家的答案进行比较.
- 评估LLM生成的答案与专家策划的信息的协调性.
主要方法:
- 5个LLM聊天机器人 (ChatGPT,Gemini,Copilot,DeepSeek,Grok-3) 得到了45个PICO问题.
- 答案与医学信息科学家生成的答案进行了比较.
- 独立的科学家将对齐分为总,部分或没有对齐.
- 分析了部分一致的答案,以提供额外的信息.
主要成果:
- 在对齐评级 (p=0.46) 中,在五个LLM聊天机器人之间没有发现显著的性能差异.
- 在LLM中,实现总对齐率为20.9%,部分对齐率为78.7%.
- 与LLM相比,医疗信息科学家在部分一致的答案中提供了显著更多的额外相关信息 (p=0.02).
结论:
- 目前领先的LLM聊天机器人在回答与医学信息科学家相关的临床问题方面表现相似.
- 虽然LLM提供了大量信息,但人类专家提供了更深入的补充细节.
- 未来的研究应该专注于验证LLM和人类专家提供的额外信息的准确性和相关性.
相关概念视频
Nursing Clinical Information System
1.2K
Nursing Clinical Information System (NCIS)
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
1.2K
Non-equilibrium in the Cell
5.3K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
5.3K
Clinical Trials: Overview
4.6K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
4.6K
Current Trends in Nursing II
3.3K
Trends in nursing are multifactorial and associated with changes in society, within the nursing profession, and in other professions. Notably, telehealth and remote nursing contribute to successful healthcare delivery for numerous patients and help reduce stress for nurses due to nursing shortages. Nurses can reach patients, monitor their conditions, and interact with them using computers, audio, visual accessories, and telephones—for example, remote patient monitoring systems. Likewise,...
3.3K
Clinical Trials
10.1K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
10.1K
Combination Therapies and Personalized Medicine
5.9K
Combining two or more treatment methods increases the life span of cancer patients while reducing damage to vital organs or tissue from the overuse of a single treatment. Combination therapy also targets different cancer-inducing pathways, thus reducing the chances of developing resistance to treatment.
The combination of the drug acetazolamide and sulforaphane is a good example of combination therapy to treat cancer. The cells in the interior of a large tumor often die due to the hypoxic and...
The combination of the drug acetazolamide and sulforaphane is a good example of combination therapy to treat cancer. The cells in the interior of a large tumor often die due to the hypoxic and...
5.9K

