从临床精神病学笔记中提取社会支持和社会隔离信息:比较基于规则的自然语言处理系统和大型语言模型
Braja Gopal Patra1, Lauren A Lepow2, Praneet Kasi Reddy Jagadeesh Kumar1
1Department of Population Health Sciences, Weill Cornell Medicine, New York, NY 10065, USA.
Journal of the American Medical Informatics Association : JAMIA
|October 18, 2024
概括
基于规则的系统 (RBS) 在从临床笔记中提取社会支持和隔离数据方面表现优于大型语言模型 (LLM). 通过严格遵守注释规则,RBS实现了更高的准确性,与LLM的更广泛解释不同.
科学领域:
- 医疗信息学 医疗信息学
- 自然语言处理自然语言处理.
- 精神病学是一个精神病学.
背景情况:
- 社会支持 (SS) 和社会隔离 (SI) 是健康 (SDOH) 的关键社会决定因素,影响精神病学结果.
- 关于SS/SI的信息通常嵌入到电子健康记录 (EHR) 中的非结构化的叙述性临床笔记中.
- 使用自然语言处理 (NLP) 从临床笔记中自动提取SS / SI数据可以简化数据收集.
研究的目的:
- 为了比较基于规则的系统 (RBS) 和大型语言模型 (LLM) 在从精神病学临床笔记中提取SS / SI信息的性能.
- 评估NLP两种方法在识别SS/SI提及及其子类别 (例如,社交网络,工具支持,孤独) 中的准确性.
主要方法:
- 通过对西奈山卫生系统 (MSHS) 和威尔康奈尔医学 (WCM) 的精神病遭遇笔记进行注释,创建了一个黄金标准的语料库.
- 使用词典的基于规则的系统 (RBS) 和基于FLAN-T5-XL的大型语言模型 (LLM) 已被开发用于SS/SI提取.
- 在SS/SI提取和子类别识别方面,使用宏观平均F1分数来评估性能.
主要成果:
- 在MSHS (0.89对0.65) 和WCM (0.85对0.82) 两个方面,RBS在提取SS/SI方面获得了比LLM更高的宏观平均F1分数.
- 对于分类提取,RBS在MSHS (0.90对0.62) 和WCM (0.82对0.81) 两种方面都超过了LLM.
结论:
- 在所有评估的社会支持和隔离数据提取指标中,RBS意外地超过了LLM.
- 这种绩效差异归因于RBS遵守特定的注释规则,而LLM更普遍的英语语言理解.
- 虽然这两种NLP方法都有优点,但建议进行进一步的复制研究以验证这些发现.
相关概念视频
Classification of Illness
9.4K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
9.4K
Automatic Processing and Automatic Social Behavior
376
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
376


