医学文档中大型语言模型的评估框架:开发和可用性研究
Junhyuk Seo1,2, Dasol Choi1, Taerim Kim1,3
1Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Korea.
Journal of medical Internet research
|November 20, 2024
概括
这项研究开发并验证了大型语言模型 (LLM) 生成的医疗记录的评估框架. 该框架可靠地评估准确性和临床适用性,支持将AI整合到医疗保健文档中.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 临床文档 临床文档
背景情况:
- 大型语言模型 (LLM) 为医疗保健文档提供了机会,但在准确性,可靠性和标准化方面面临挑战.
- 由于质量保证缺陷,对LLM生成的医疗记录的临床应用存在担忧.
研究的目的:
- 开发和验证一个评估框架,以评估LLM生成的急诊室 (ED) 记录的准确性和临床适用性.
- 通过强大的评估系统,加强人工智能在医疗保健文档中的整合.
主要方法:
- 一个竞争性活动,医疗保健Prompt-a-thon,涉及52名参与者使用HyperCLOVA X.生成33个ED记录.
- 使用双重评估方法:由4名医疗专业人员进行的临床评估 (利克尔特尺度) 和定量错误分析 (7种错误类型).
- 统计方法,包括皮尔森相关性和类内相关系数 (ICC),用于可靠性和协议评估.
主要成果:
- 临床评估显示了强大的评分器间可靠性 (ICC 0.653-0.887) 和测试重试可靠性 (皮尔森r=0.776).
- 无效生成错误是最常见的 (35.38%),而结构错误对临床评分的影响最为负面 (Pearson r=-0.654).
- 在定量错误和临床评估得分之间发现了显著的负相关性 (Pearson r=-0.633),这表明错误较高的可接受性较低.
结论:
- 拟议的评估框架可靠且在临床上可接受,用于评估LLM生成的ED记录.
- 该框架可以减轻临床负担,并促进负责任的AI整合到医疗保健中.
- 这项研究为医疗文档中未来的人工智能应用提供了有希望的方向.
相关概念视频
Guidelines for Nursing Documentation I
1.0K
Quality documentation and reporting share essential characteristics that ensure they are practical and valuable resources for those who use them. These characteristics are:
Factual:
The following points emphasize the significance of upholding accurate and unbiased documentation in healthcare.
Factual:
The following points emphasize the significance of upholding accurate and unbiased documentation in healthcare.
1.0K
Methods of Documentation VI: Case Management Model
559
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
559
Introduction to Documentation and Reporting
1.9K
Documentation is the systematic process of formally recording, maintaining, and communicating information.
Nursing documentation records essential information and details regarding a patient's care and treatment in written or electronic form. It is a critical aspect of nursing practice that involves documenting assessments, interventions, outcomes, and other relevant details about a patient's health status.
Documentation maps the patient's health journey by creating a comprehensive...
Nursing documentation records essential information and details regarding a patient's care and treatment in written or electronic form. It is a critical aspect of nursing practice that involves documenting assessments, interventions, outcomes, and other relevant details about a patient's health status.
Documentation maps the patient's health journey by creating a comprehensive...
1.9K
Methods of Documentation V: CBE
882
Charting by Exception, or CBE, is a method of documentation used in healthcare, particularly in nursing, that focuses on documenting only significant or abnormal findings rather than recording every detail. This approach aims to streamline the documentation process, improve efficiency, and ensure that healthcare providers can quickly identify deviations from normalcy in patient assessments.
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
882
Documentation in Long-Term and Home Healthcare Setting
873
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
873
Methods of Documentation II: POMR
914
The Problem-Oriented Medical Record (POMR) revolutionized medical record-keeping by introducing a systematic approach focusing on the patient's problems rather than merely listing symptoms. Dr. Lawrence Weed's introduction of this method in the 1960s marked a significant advancement in medical documentation. The POMR framework consists of four key components: the database, problem list, plan of care, and progress notes.
914


