使用多样化测试套件来评估快速医疗保健互操作性资源的大型语言模型知识:比较分析
Ahmad Idrissi-Yaghir1,2, Kamyar Arzideh2,3, Henning Schäfer2,4
1Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
Journal of medical Internet research
|August 12, 2025
概括
大型语言模型 (LLM) 在快速医疗互操作资源 (FHIR) 任务中表现出色,像GPT-4o这样的商业模型表现出色. 然而,将临床笔记转换为FHIR仍然是所有模型的挑战.
科学领域:
- 人工智能的人工智能
- 医疗信息学 医疗信息学
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 在一般知识方面表现出先进的能力,但在专门的快速医疗互操作资源 (FHIR) 标准中评估有限.
- 对于在广泛数据集上接受培训的法学士来说,FHIR的复杂性带来了挑战,可能会限制他们对特定领域的医疗保健数据的理解.
- 加强健康数据的互操作性对于利用临床数据和改善电子健康记录交互至关重要.
研究的目的:
- 介绍FHIR工作台,这是一个新的数据集套件,旨在严格评估LLM理解和应用FHIR标准.
- 评估开源和商业LLM在各种FHIR相关任务上的表现.
- 为了解医疗保健数据互操作性的LLM能力提供一个基准.
主要方法:
- 开发了四个不同的评估数据集来测试FHIR知识,包括关于FHIR概念和FHIR代表状态转移 (REST) API的多选择题.
- 在零射击环境下评估LLM的表现,例如识别FHIR资源类型和从非结构化临床笔记中生成FHIR资源.
- 包括与六名不同FHIR专业知识的参与者进行的人类评估,以将LLM绩效指标置于背景中.
主要成果:
- 像GPT-4o这样的商业模型获得了高的F1分数 (例如,FHIR-ResourceID上的0.9990),而像DeepSeek-v3这样的开源模型也显示了强的结果 (例如,FHIR-QA上的0.9400).
- 所有评估的LLM都在Note2FHIR任务中遇到了困难,这表明在将非结构化临床文本转换为符合FHIR资源方面存在重大挑战,得分范围从0.0382到0.3633.
- 人类参与者在最初的三个FHIR任务中获得了0.50到1.0之间的准确度得分.
结论:
- 开源和商业LLM都在FHIR相关任务上表现出竞争力的表现,商业模型目前在复杂的应用中处于领先地位.
- FHIR工作台是评估LLM能力和推动健康数据互操作性进步的关键工具.
- 需要持续发展,以应对LLM在准确处理和将非结构化临床数据转换为标准化FHIR资源方面所面临的挑战.
相关概念视频
Multiple Comparison Tests
4.0K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
4.0K
Documentation in Long-Term and Home Healthcare Setting
985
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
985
Methods of Documentation VI: Case Management Model
628
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
628
Health Information Technology and Healthcare Information System
917
Health Information Technology (HIT)
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
917
Nursing Clinical Information System
865
Nursing Clinical Information System (NCIS)
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
A Nursing Clinical Information System (NCIS) is a specialized type of healthcare information system tailored to meet the unique needs of nursing practice. It incorporates the principles of nursing informatics to streamline information management and improve the quality of care delivery.
Critical attributes of NCIS include:
865
Methods Of Healthcare Delivery System
3.5K
At the different levels of the healthcare system, we see varying methods of healthcare used. These methods include managed care systems, case management, and primary healthcare.
Managed Care System:
The managed care system is designed to control the cost while maintaining the quality of care. The patient's care from admission to discharge is planned by the primary care provider or the case manager, also known as the gatekeeper. In a managed care system, the number of care providers is...
Managed Care System:
The managed care system is designed to control the cost while maintaining the quality of care. The patient's care from admission to discharge is planned by the primary care provider or the case manager, also known as the gatekeeper. In a managed care system, the number of care providers is...
3.5K


