在预测人工智能模型中对性能指标的评估,以支持医疗决策:概述和指导
Ben Van Calster1, Gary S Collins2, Andrew J Vickers3
1Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands; Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands.
选择正确的绩效指标对于医学人工智能 (AI) 模型至关重要. 这项研究评估了对预测二进制结果的AI模型的32项措施,突出了临床验证的重要指标.
科学领域:
- 医疗人工智能 医疗人工智能
- 临床决策支持系统 临床决策支持系统
- 预测分析在医疗保健中的应用
背景情况:
- 选择适当的绩效指标对于在医疗实践中验证预测人工智能 (AI) 模型至关重要.
- 选择不当的指标可能会导致误导人工智能性能,可能导致错误的临床决策,患者伤害和成本增加.
研究的目的:
- 评估经典和当代性能指标的优点,以验证估计二进制结果概率的AI模型.
- 讨论五个领域的32个绩效指标:歧视,校准,整体性能,分类和临床实用性,包括图形评估.
- 突出选择绩效指标的关键特征:它们是否"正确",并考虑错误分类成本.
主要方法:
- 对预测二进制结果的AI模型的32个性能指标的评估.
- 将措施分为五个性能领域的分类:歧视,校准,整体性能,分类和临床实用性.
- 评估基于两个关键特征的措施:是"适当的"和考虑决策分析绩效 (错误分类成本).
主要成果:
- 在32项措施中,17项具有两个理想的特征 (适当性和错误分类成本的核算),14项具有一个,而F1评分则没有任何一项.
- 对于临床相关的决定值,除了0.5或真实流行率之外,被发现分类措施不合适.
- 用于预测卵巢瘤恶性瘤的ADNEX模型被用于说明这些措施及其特征.
结论:
- 建议在临床环境中报告AI模型性能的基本措施和图表.
- 基本报告包括:接收器运行特征曲线下的面积,校准图,临床效用指标 (例如,净效益与决策曲线分析) 和概率分布图.
- 强调选择适当的措施的重要性,以考虑决策分析性能,以便在医学中可靠地验证AI模型.
相关概念视频
Issues And Trends In Healthcare Delivery System
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
Measures of Intelligence
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
Sensitivity, Specificity, and Predicted Value
Sensitivity is the...
Methods of Documentation VI: Case Management Model
For example, a patient with a chronic...
Errors occurring during blood pressure monitoring
Several factors...
Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...

