对医疗问答任务的大型语言模型响应的实体中心评估
Yi Liu1, Vijaya B Kolachalama1,2,3
1Faculty of Computing & Data Sciences, Boston University, MA, USA.
medRxiv : the preprint server for health sciences
|November 26, 2025
概括
一个新的指标,EntQA,通过专注于临床准确性和实体保留,有效地评估医学问答中的大语言模型 (LLM) 响应. 该工具通过改进事实核查和模型改进来增强可信的医疗保健AI.
科学领域:
- 人工智能在医学中的应用
- 自然语言处理自然语言处理.
- 生物医学信息学 生物医学信息学
背景情况:
- 在医学问答 (QA) 中评估大型语言模型 (LLM) 响应的临床准确性对于安全部署至关重要.
- 现有的指标可能无法充分捕捉LLM产生的医疗建议中临床调整和信息性的细微差别.
研究的目的:
- 开发和验证EntQA,一种新的以实体为中心的指标,用于评估医学QA中LLM输出的临床忠实性.
- 量化EntQA在不依赖外部知识库的情况下测量实体保留和检测遗漏的能力.
主要方法:
- 拟议的EntQA指标,利用生物医学命名实体识别 (NER) 来从患者数据,问题和LLM响应中提取和匹配实体.
- 使用除重复和语义/词汇匹配与定义的值.
- 计算了回忆式覆盖分数,并使用7个不同尺寸的Qwen 2.5 Instruct模型对5个基准进行了性能评估.
主要成果:
- EntQA在组 (斯皮尔曼高达0.9286) 和病例水平 (点-双序列高达0.0926) 上都显示出与模型准确性的强烈正相关性.
- 该指标显示与模型缩放 (斯皮尔曼高达0.252) 一致一致.
- EntQA的表现优于基线指标,其中一些表现为负或不一致的准确性相关性.
结论:
- EntQA提供了一个可扩展和可解释的方法来评估基于LLM的医疗质量保证系统.
- 该指标在捕捉临床准确性方面表现出色,超过了传统的评估方法.
- EntQA支持通过事实检查和模型改进的应用程序开发可靠的医疗保健AI.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
1.3K
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
16.4K
相关概念视频
Patient-centered Care
2.9K
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
2.9K
Classification of Illness
8.5K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
8.5K
Language and Cognition
696
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
696
Specialized Care Centers and Settings-II
1.2K
Rural Health Centers
Rural health centers are specialized care facilities in remote locations with very few medical personnel. The primary care providers who run the centers are mostly Registered Nurse Practitioners. Here, emergency treatment is provided to critically ill or injured patients before they are transferred to the closest hospital. Fortunately, due to advancement in technology, many rural healthcare facilities and professionals have easy access to diagnostic and treatment...
Rural health centers are specialized care facilities in remote locations with very few medical personnel. The primary care providers who run the centers are mostly Registered Nurse Practitioners. Here, emergency treatment is provided to critically ill or injured patients before they are transferred to the closest hospital. Fortunately, due to advancement in technology, many rural healthcare facilities and professionals have easy access to diagnostic and treatment...
1.2K
Integrated Healthcare System
2.3K
An integrated healthcare system (IHS) is a set of organizations that provides for or arranges to provide coordinated and continuous service to a defined population. The IHS takes responsibility for that particular population's health status and outcome, both clinically and fiscally. An integrated healthcare system is a well-organized, well-coordinated, and collaborative network. The integrated delivery system is a network that connects different healthcare providers to deliver organized,...
2.3K
Nursing Evaluation
4.1K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
4.1K
