开发和验证大型语言模型的提供者文档总结质量工具
Emma Croxford1, Yanjun Gao2, Nicholas Pellegrino3
1Department of Biostatistics and Medical Informatics, University of Wisconsin, Madison, WI 53792, United States.
概括
提供者文档总结质量工具-9 (PDSQI-9) 是一个验证的工具,用于评估大型语言模型 (LLM) 临床总结. 它确保对LLM生成的文档进行可靠的评估,以便更安全地整合医疗保健.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 临床文档 临床文档
背景情况:
- 大型语言模型 (LLM) 越来越多地被整合到电子健康记录 (EHR) 工作流中.
- 经过验证的工具对于评估医疗保健环境中的LLM绩效至关重要.
- 现有的文档质量工具往往对LLM生成的文本不够.
研究的目的:
- 验证提供者文档总结质量工具-9 (PDSQI-9) 用于评估LLM生成的临床总结.
- 在多个维度中评估 PDSQI-9 的构造有效性.
- 提供一个可靠的工具来评估LLM在EHR文档中的表现.
主要方法:
- 从现实世界EHR数据中生成多文件摘要,使用各种LLM.
- 使用皮尔森相关性,因子分析,克朗巴赫的α,评分器间可靠性 (ICC,Krippendorff的α),半德尔菲过程和判别性有效性测试验证实了PDSQI-9.
- 医生评级人员接受了标准化的培训,以确保一致的仪器应用.
主要成果:
- PDSQI-9表现出强大的内部一致性 (克朗巴赫的α=0.879) 和高的评分间可靠性 (ICC=0.867).
- 因子分析揭示了一个4个因素模型 (组织,清晰度,准确性,实用性),解释了58%的差异.
- 该工具显示了良好的实质有效性,内容相关性和区分有效性,区分高质量和低质量的摘要 (P<.001).
结论:
- PDSQI-9表现出强大的构造有效性,高可靠性和有意义的因子结构,用于评估LLM生成的临床摘要.
- 该工具可靠地捕获文档质量的关键维度,并区分高质量和低质量的摘要.
- PDSQI-9是卫生系统的一种实际工具,以确保更安全,更有效地将LLM纳入临床工作流程.
关键词:
临床自然语言生成电子健康记录 电子健康记录评价 评价 评价 评价生成型的人工智能 (generative artificial intelligence) 是一种创造性的人工智能.大型语言模型.多个文件的总结.自然语言处理自然语言处理.更多相关视频
相关概念视频
Improving Translational Accuracy
2.5K
2.5K
Methods of Documentation VI: Case Management Model
557
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
557
Introduction to Documentation and Reporting
1.9K
Documentation is the systematic process of formally recording, maintaining, and communicating information.
Nursing documentation records essential information and details regarding a patient's care and treatment in written or electronic form. It is a critical aspect of nursing practice that involves documenting assessments, interventions, outcomes, and other relevant details about a patient's health status.
Documentation maps the patient's health journey by creating a comprehensive...
Nursing documentation records essential information and details regarding a patient's care and treatment in written or electronic form. It is a critical aspect of nursing practice that involves documenting assessments, interventions, outcomes, and other relevant details about a patient's health status.
Documentation maps the patient's health journey by creating a comprehensive...
1.9K
Documentation in Long-Term and Home Healthcare Setting
868
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
868
Methods of Documentation II: POMR
894
The Problem-Oriented Medical Record (POMR) revolutionized medical record-keeping by introducing a systematic approach focusing on the patient's problems rather than merely listing symptoms. Dr. Lawrence Weed's introduction of this method in the 1960s marked a significant advancement in medical documentation. The POMR framework consists of four key components: the database, problem list, plan of care, and progress notes.
894
Guidelines for Nursing Documentation I
1.0K
Quality documentation and reporting share essential characteristics that ensure they are practical and valuable resources for those who use them. These characteristics are:
Factual:
The following points emphasize the significance of upholding accurate and unbiased documentation in healthcare.
Factual:
The following points emphasize the significance of upholding accurate and unbiased documentation in healthcare.
1.0K


