大型语言模型编码临床知识
Karan Singhal1, Shekoofeh Azizi2, Tao Tu3
1Google Research, Mountain View, CA, USA. karansinghal@google.com.
Nature
|July 12, 2023
概括
大型语言模型 (LLM) 在医学上是有前途的,但需要严格评估. 一个新的基准,MultiMedQA和人类评估揭示了当前LLM的局限性,强调了改善临床AI发展的需要.
科学领域:
- 人工智能
- 医疗信息学
- 自然语言处理
背景情况:
- 大型语言模型 (LLM) 具有先进的功能,但在临床使用中面临着高标准.
- 目前对LLM医学知识的评估通常依赖于有限的自动化基准.
研究的目的:
- 推出MultiMedQA,一个全面的衡量标准来评估医疗问答中的LLM.
- 建立一个人类评估框架,评估LLM响应中的事实性,理解,推理,伤害和偏见.
- 评估Pathways语言模型 (PaLM) 和Flan-PaLM在MultiMedQA基准上的表现.
主要方法:
- 开发了MultiMedQA,整合了六个医疗QA数据集和新的HealthSearchQA数据集.
- 实施了对LLM产生的医疗答案的人类评估协议.
- 在MultiMedQA上使用各种提示策略评估PaLM和Flan-PaLM.
- 引入指令提示符调整以适应LLM的领域.
主要成果:
- 在所有MultiMedQA多选题数据集中,Flan-PaLM实现了最先进的精度,其中MedQA (USMLE样式问题) 的精度为67.6%.
- 人类评估发现,尽管自动化得分很高,但LLM的表现存在重大差距.
- 指示提示调整导致Med-PaLM,其表现有所改善,但仍低于临床医生的水平.
结论:
- 医学LLM的表现会随着尺度和教学提示的调整而提高.
- 目前的LLM在临床应用方面存在局限性,这凸显了对强有力的评估框架的需求.
- 进一步发展对于为医疗保健创造安全有效的LLM至关重要.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
318
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
16.0K
相关概念视频
Methods of Documentation VI: Case Management Model
598
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
598
Clinical Trials: Overview
3.0K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
3.0K
Nursing Clinical Information System
823
Nursing Clinical Information System (NCIS)
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
823
Higher Mental Functions of the Brain: Language
923
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
923
Classification of Illness
7.6K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
7.6K
Clinical Trials
6.9K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
6.9K
