在医学证据总结和总结上评估大型语言模型
Liyan Tang1, Zhaoyi Sun2, Betina Idnay3
1School of Information, The University of Texas at Austin, Austin, TX, USA.
NPJ digital medicine
|August 24, 2023
概括
大型语言模型 (LLM) 在医学总结方面表现有前途,但在事实一致性和确定关键信息方面存在困难,造成错误信息的风险.
科学领域:
- 人工智能的人工智能
- 医疗信息学 医疗信息学
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 在各个任务中表现出强大的零和少数射击能力.
- 医疗保健等高风险领域正在探索LLM应用.
研究的目的:
- 在零射击医学证据总结中系统评估LLM绩效 (GPT-3.5,ChatGPT).
- 通过在六个临床领域的自动和人体评估来评估总结质量.
主要方法:
- 对LLM生成的医学摘要进行了自动和人为评估.
- 定义了基于人类评估的错误类型的分类.
- 分析了不同临床领域和文本长度的摘要质量.
主要成果:
- 自动指标显示与人类判断的摘要质量有很弱的相关性.
- 实际上,LLM产生的摘要与事实不一致,并表现出有问题的确定性水平.
- 模型难以识别突出信息,较长文本的准确性较低.
结论:
- 由于可能存在事实错误和错误信息,LLM需要对医学证据总结进行仔细的验证.
- 人类评估对于评估LLM生成的医学摘要的质量和安全性至关重要.
- 处理长文本和识别关键信息的局限性需要进一步研究.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
277
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
15.9K
相关概念视频
Methods of Documentation VI: Case Management Model
598
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
598
Improving Translational Accuracy
2.6K
2.6K
Health Literacy
4.1K
Health literacy is an individual's or a community's capacity to comprehend, receive, read, and use relevant healthcare information and services. The World Health Organization (WHO, 2018) defines health literacy as the cognitive and social skills that determine the ability of individuals to gain access to, understand, and use information in ways that promote and maintain good health. As a result, the WHO helps individuals manage long-term health concerns, participate in preventative...
4.1K
Clinical Trials: Overview
3.0K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
3.0K
