开发人类评估框架和与自然语言自动化指标的相关性 医学诊断的生成
Emma Croxford1, Yanjun Gao1, Brian Patterson2
1Department of Medicine, School of Medicine and Public Health, University of Wisconsin Madison.
medRxiv : the preprint server for health sciences
|April 2, 2024
概括
自动化指标难以评估临床自然语言生成 (NLG) 质量. 特定领域的知识,如SapBERT评分,对于准确的医疗保健文本评估至关重要.
科学领域:
- 医疗信息学 医疗信息学
- 自然语言处理自然语言处理.
背景情况:
- 评估临床自然语言生成 (NLG) 文本质量是很困难的.
- 现有的自动化指标往往无法捕捉医疗保健中生成任务的复杂性.
研究的目的:
- 检查医疗保健领域NLG自动化评估指标的现状.
- 建立一个全面的人类评估框架,作为评估指标绩效的基准.
- 确定各种自动化指标与临床NLG输出的人类判断的对齐.
主要方法:
- 开发了一个全面的人类评估框架来评估抽象文本的质量.
- 使用ChatGPT-3.5-turbo生成临床文本输出.
- 人类对生成文本的判断与几个自动评估指标相关联,包括使用统一医疗语言系统 (UMLS) 的SapBERT得分.
主要成果:
- 没有任何自动化指标表明与临床NLG文本质量的人类判断高度一致.
- 纳入UMLS领域特定医学知识的SapBERT评分,在评估的指标中表现最好.
- 这表明当前对生成的临床文本的自动化质量评估存在重大缺陷.
结论:
- 目前临床NLG的自动化评估指标缺乏.
- 拟议的人类评估框架作为一个强大的基线.
- 未来的工作应该集中在整合医疗知识数据库,以改进像SapBERT这样的指标,以提高准确性.
相关概念视频
Data Validation
5.0K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
5.0K
Formulating and Validating Nursing Diagnosis I
2.6K
A nursing diagnosis is written when the nurse recognizes a cluster of essential patient data indicating health problems treated with independent nursing interventions. The standardized terminologies of a nursing diagnosis help nurses identify and treat patients' problems. Every electronic health record that uses nursing diagnosis must employ standard diagnostic terminology. Developing an efficient, individualized care plan begins with accurate nursing diagnoses.
There are thirteen domains...
There are thirteen domains...
2.6K
Nursing Evaluation
3.3K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.3K
Documentation of Nursing Diagnosis
1.2K
The nurse documents nursing diagnoses and enters them into the patient record. The identified patient's nursing diagnosis is either written out with a plan of care or entered into the electronic health record.
In some settings, data-driven computerized decision support systems are in place, allowing for more accurate nursing diagnoses. The database within one of these systems includes diagnostic labels defining characteristics, activities, and indicators for nursing. A nurse enters...
In some settings, data-driven computerized decision support systems are in place, allowing for more accurate nursing diagnoses. The database within one of these systems includes diagnostic labels defining characteristics, activities, and indicators for nursing. A nurse enters...
1.2K


