大型语言模型对医疗保健应用的测试和评估:一个系统性审查
Suhana Bedi1, Yutong Liu2, Lucy Orr-Ewing2
1Department of Biomedical Data Science, Stanford School of Medicine, Stanford, California.
JAMA
|October 15, 2024
概括
目前对医疗保健中的大型语言模型 (LLM) 的评估主要是对考试的准确性进行评估, 未来的研究需要更广泛的范围和临床相关性.
科学领域:
- 医疗保健中的人工智能
- 自然语言处理 (NLP)
- 医疗信息学
背景情况:
- 大型语言模型 (LLM) 显示出医疗应用的潜力.
- 现有的评估方法可能无法充分体现LLM在临床环境中的有用性.
- 需要全面了解当前的LLM评估,以指导未来的研究.
研究的目的:
- 系统地审查和分类医疗保健中的现有LLM评估.
- 根据数据类型,医疗保健任务,NLP/NLU任务,评估维度和医疗专业来分析评估.
- 确定医学领域当前LLM评估策略中的差距和局限性.
主要方法:
- 在PubMed和Web of Science的系统文献搜索 (2022年1月 - 2024年2月).
- 包括评估医疗保健中的一个或多个LLM的研究.
- 独立评审人员对519项研究进行了分类,分别对5个主要组成部分进行了分类.
主要成果:
- 只有5%的研究使用了实际的患者护理数据;大多数专注于医学知识评估 (例如,执照考试) 和诊断.
- 回答问题是NLP/NLU的主要任务 (84.2%),仅限于总结或对话.
- 准确性是主要评估指标 (95.4%),而公平性,偏见,毒性和部署考虑因素很少被评估.
结论:
- 目前在医疗保健领域的LLM评估严重偏向于标准化测试的准确性,缺乏真实世界的临床数据和多样化的任务考虑.
- 公平性,偏见和部署准备等关键方面被低估.
- 未来的研究应优先考虑标准化指标,临床数据利用,以及更广泛的医疗保健任务和专业,以进行强有力的LLM评估.
相关概念视频
Issues And Trends In Healthcare Delivery System
5.6K
The issues and trends in healthcare delivery are constantly changing. The COVID-19 pandemic is one recent issue that wreaked havoc on healthcare systems, causing a shortage of healthcare workers, high demand for medicines and supplies, and increased medical expenditure due to a lack of insurance. Other issues include rising healthcare costs and care fragmentation.
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
5.6K
Methods of Documentation VI: Case Management Model
559
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
559
Health Information Technology and Healthcare Information System
806
Health Information Technology (HIT)
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
806
Documentation in Long-Term and Home Healthcare Setting
873
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
873
Statistical Software for Data Analysis and Clinical Trials
502
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
502
Purpose of Health Records I
1.2K
The vital purpose of health records is to provide a complete and accurate account of a patient's medical history, including communication, diagnostic and therapeutic orders, care planning, research, and quality review.
Here's a breakdown of how health records serve these purposes:
Here's a breakdown of how health records serve these purposes:
1.2K


