对医疗保健的可重复生成人工智能评估:一个临床医生在循环中的方法
Leah Livingston1, Amber Featherstone-Uwague1, Amanda Barry1
1Elsevier, Health Markets, Philadelphia, PA 19103, United States.
JAMIA open
|June 17, 2025
概括
一个新的框架评估了医疗保健中的生成人工智能 (AI),显示出高准确性和有用性. 这种可重复的方法支持在临床环境中安全地实施AI.
科学领域:
- 医疗信息学 医疗信息学
- 人工智能在医学中的应用
- 临床评估方法 临床评估方法
背景情况:
- 生成型人工智能 (AI) 系统在医疗保健中越来越多地使用,但实际的评估方法落后于理论框架.
- 需要可重复的方法来评估AI工具在临床实践中的性能和安全性.
研究的目的:
- 开发和应用可重复的方法来评估医疗保健中的生成人工智能系统.
- 解决理论AI评估框架和实际实施指南之间的差距.
主要方法:
- 创建了一个5维评估框架,以评估AI查询理解和响应质量 (有用性,正确性,完整性,潜在的临床危害).
- 该框架应用于ClinicalKey AI,由41名医生和药剂师使用来自不同来源的查询进行评估.
- 一个协议协议,包括修改的Delphi方法,用于解决评估分数的分歧.
主要成果:
- 在633个查询中,96.99%的查询得到了可评估的答案,其中426对被专家评估.
- 观察到高性能:查询理解率为98.6%,答案正确率为95.5%,帮助率为94.4%.
- 确定了潜在临床危害的最小率 (0.47%),强调了严格评估的重要性.
结论:
- 开发的框架有效量化了医疗保健中的生成AI性能,使用了全面的维度和结构化的评分.
- 优势包括多样化的查询抽样,标准化的评级和专家协议协议,尽管主观评估和危害分类仍然存在挑战.
- 这种可重复的方法提供了评估和在临床环境中实施生成AI的基础过程.
相关概念视频
01:29Issues And Trends In Healthcare Delivery System
5.9K
The issues and trends in healthcare delivery are constantly changing. The COVID-19 pandemic is one recent issue that wreaked havoc on healthcare systems, causing a shortage of healthcare workers, high demand for medicines and supplies, and increased medical expenditure due to a lack of insurance. Other issues include rising healthcare costs and care fragmentation.
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
5.9K
01:20Integrated Healthcare System
1.8K
An integrated healthcare system (IHS) is a set of organizations that provides for or arranges to provide coordinated and continuous service to a defined population. The IHS takes responsibility for that particular population's health status and outcome, both clinically and fiscally. An integrated healthcare system is a well-organized, well-coordinated, and collaborative network. The integrated delivery system is a network that connects different healthcare providers to deliver organized,...
1.8K
01:15Nursing Evaluation
3.6K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.6K
01:26Methods Of Healthcare Delivery System
3.5K
At the different levels of the healthcare system, we see varying methods of healthcare used. These methods include managed care systems, case management, and primary healthcare.
Managed Care System:
The managed care system is designed to control the cost while maintaining the quality of care. The patient's care from admission to discharge is planned by the primary care provider or the case manager, also known as the gatekeeper. In a managed care system, the number of care providers is...
Managed Care System:
The managed care system is designed to control the cost while maintaining the quality of care. The patient's care from admission to discharge is planned by the primary care provider or the case manager, also known as the gatekeeper. In a managed care system, the number of care providers is...
3.5K
01:17Healthcare Agencies II
760
There are various healthcare agencies in the United States—some of which are managed by religious institutions and others by different government branches.
Parish nursing is a growing specialty nursing profession that focuses on holistic healthcare, health promotion, and illness prevention. It blends professional nursing practice with a health ministry, focusing on health and healing within the context of a Christian community. Parish nurses serve as health educators, referral sources,...
Parish nursing is a growing specialty nursing profession that focuses on holistic healthcare, health promotion, and illness prevention. It blends professional nursing practice with a health ministry, focusing on health and healing within the context of a Christian community. Parish nurses serve as health educators, referral sources,...
760
01:13Patient-centered Care
2.3K
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
2.3K