医疗保健中大型语言模型的人类评估的框架,来源于文献审查
Thomas Yu Chow Tam1, Sonish Sivarajkumar2, Sumit Kapoor3
1Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA, USA.
NPJ digital medicine
|September 27, 2024
概括
人类对生成人工智能 (GenAI) 的评估,特别是医疗保健中的大型语言模型 (LLM),至关重要. 本研究确定了当前方法中的差距,并提出了QUEST,一个新的框架,以确保安全有效的LLM部署.
科学领域:
- 医疗信息学 医疗信息学
- 人工智能在医学中的应用
背景情况:
- 生成型人工智能 (GenAI),特别是大型语言模型 (LLM),越来越多地用于医疗保健.
- 通过人类评估来评估LLM对于确保其在临床环境中的安全性和有效性至关重要.
研究的目的:
- 对医疗保健中LLM的人类评估方法的现有文献进行审查.
- 识别有关可靠性,通用性和适用性的当前评估实践中的差距.
- 提出一个全面的框架,QUEST,用于对医疗保健LLMs的人类评估.
主要方法:
- 一个文献综述142项关于在医疗保健中LLMs的人类评估方法的研究.
- 分析因素,包括评估尺寸,样本特征,评估者选择,框架,指标,流程和统计分析.
- 基于已识别的差距和最佳实践,开发QUEST框架.
主要成果:
- 文献综述揭示了当前医疗保健LLM人体评估实践的可靠性,通用性和适用性的重大差距.
- 现有的方法通常缺乏评估维度,评估员招聘和统计分析的标准化.
- 拟议的QUEST框架通过提供结构化的方法来解决这些局限性.
结论:
- 目前用于医疗保健LLM的人类评估方法不足,阻碍了安全的开发和部署.
- QUEST框架为强大的LLM评估提供了一个全面而实用的解决方案.
- 实施QUEST及其五个核心原则,将提高医疗保健LLM的质量和可信度.
相关概念视频
Models of Health Promotion and Illness Prevention I
2.0K
A model is a theoretical way to understand a concept or an idea. Models can overcome barriers to health regardless of diverse economic and cultural backgrounds. In addition, models make the task easier by providing different ways to approach complex issues. There are two major health promotion models: the health belief model and the health promotion model.
The health belief model (HBM) attempts to predict health-related behavior in specific belief patterns. According to the HBM, a person's...
The health belief model (HBM) attempts to predict health-related behavior in specific belief patterns. According to the HBM, a person's...
2.0K
Methods of Documentation VI: Case Management Model
559
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
559
Health Literacy
4.0K
Health literacy is an individual's or a community's capacity to comprehend, receive, read, and use relevant healthcare information and services. The World Health Organization (WHO, 2018) defines health literacy as the cognitive and social skills that determine the ability of individuals to gain access to, understand, and use information in ways that promote and maintain good health. As a result, the WHO helps individuals manage long-term health concerns, participate in preventative...
4.0K
Documentation in Long-Term and Home Healthcare Setting
873
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
873
Data Validation
5.0K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
5.0K
Nursing Evaluation
3.3K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.3K


