从临床文档中提取国际疾病分类代码的大型语言模型的基准测试
medRxiv : the preprint server for health sciences
|November 28, 2024
概括
大型语言模型 (LLM) 在从住院病人的笔记中提取国际疾病分类第十版 - 临床修改 (ICD-10-CM) 代码方面表现不佳. 在临床文档中,人类编码器在准确的ICD-10-CM代码提取方面仍然优越.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 改善临床文档的改善
背景情况:
- 准确的国际疾病分类-第十版 - 临床修改 (ICD-10-CM) 代码提取对于医疗补偿和编码至关重要.
- 从临床文档中自动提取ICD-10-CM代码面临着重大挑战.
- 这项研究解决了对改进自动编码解决方案的需求.
研究的目的:
- 评估各种大型语言模型 (LLM) 在从非结构化的住院临床笔记中提取ICD-10-CM代码的性能.
- 将LLM的准确性与人类医疗编码器进行基准测试.
- 确定当前LLMs在这个特定任务中的优点和弱点.
主要方法:
- 将6个LLM (GPT-3.5,GPT-4,Claude 2.1,Claude 3,Gemini Advanced,Llama 2-70b) 与一个人类编码器进行比较.
- 使用了美国健康信息管理协会Vlab真实患者病例的非身份化住院病人的笔记.
- 采用了标准化的LLM代码提取提示符和一个3M编码器,使用2022 ICD-10-CM编码指南为人类编码器.
主要成果:
- 分析了50个住院病人的笔记 (23个H&P,27个进展笔记).
- 人类编码器在每个笔记中平均识别了4个ICD-10-CM代码.
- 每个音符中,LLM 提取了 5-11 个代码的中位数,其中 GPT-4 显示出最佳性能,但与人类编码器的总体一致性仅为 15.2%.
结论:
- 当前的大型语言模型在与人类编码器相比,在从住院患者的临床笔记中提取ICD-10-CM代码时表现不充分.
- 需要进一步的进步来提高自动化临床编码的LLM准确性.
- 人类专业知识对于可靠的ICD-10-CM代码分配至关重要.
相关概念视频
01:23Methods of Documentation V: CBE
882
Charting by Exception, or CBE, is a method of documentation used in healthcare, particularly in nursing, that focuses on documenting only significant or abnormal findings rather than recording every detail. This approach aims to streamline the documentation process, improve efficiency, and ensure that healthcare providers can quickly identify deviations from normalcy in patient assessments.
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
882
01:12Statistical Software for Data Analysis and Clinical Trials
494
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
494
01:26Formulating and Validating Nursing Diagnosis I
2.6K
A nursing diagnosis is written when the nurse recognizes a cluster of essential patient data indicating health problems treated with independent nursing interventions. The standardized terminologies of a nursing diagnosis help nurses identify and treat patients' problems. Every electronic health record that uses nursing diagnosis must employ standard diagnostic terminology. Developing an efficient, individualized care plan begins with accurate nursing diagnoses.
There are thirteen domains...
There are thirteen domains...
2.6K
01:15Methods of Documentation VI: Case Management Model
559
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
559
01:29Nursing Interventions II: Selecting and Classifying the Nursing Interventions
2.1K
Creating and executing a nursing diagnosis helps nurses plan care and guide patient, family, and community interventions. They are developed based on a patient's physical evaluation and support measuring the outcomes. It is not recommended to select random interventions throughout the planning process. Instead, consider the following six essential factors when choosing interventions:
2.1K