检测临床机器学习工具的性能随时间变化而发生的变化
Michiel Schinkel1, Anneroos W Boerman2, Ketan Paranjape3
1Center for Experimental and Molecular Medicine (CEMM), Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands; Division of Acute Medicine, Department of Internal Medicine, Amsterdam UMC, VU University, Amsterdam, the Netherlands.
EBioMedicine
|October 4, 2023
概括
一个用于预测急诊室血液培养结果的机器学习模型在一年内表现稳定,尽管患者群体和临床实践发生了变化. 使用统计过程控制图表的持续监测证实了其对诊断管理的可靠性.
科学领域:
- 临床信息学 临床信息学
- 医疗保健中的机器学习
- 诊断管理管理的诊断管理.
背景情况:
- 在急诊室 (ED) 过度使用血培养 (BCs) 会导致诊断产量低,污染率高.
- 这有助于增加抗生素的使用和不必要的诊断程序.
- 一个先前开发的机器学习 (ML) 模型旨在预测BC结果并改善诊断管理.
研究的目的:
- 实时评估ML模型在预测血液培养结果方面的性能.
- 评估模型的稳定性,并识别随着时间的推移潜在的性能漂移.
- 确定需要重新校准或纠正BC管辖工具的需要.
主要方法:
- 该ML模型被集成到电子健康记录系统中,以实时预测成年ED患者的BC结果.
- 每个月使用曲线下的面积 (AUC),精度回忆曲线下的面积 (AUPRC) 和Brier分数监测模型性能.
- 使用统计过程控制 (SPC) 图表来跟踪性能变化和检测漂移.
主要成果:
- 该模型在3,035次患者访问中实现了平均AUC0.78,AUPRC0.41,Brier分数0.10.
- 尽管患者人口统计和临床实践发生了变化,但SPC图表显示模型性能稳定,没有控制范围之外的点.
- 在评估期间,平均血液培养阳性率为13.4%.
结论:
- 该BC管理工具表现出强大而稳定的表现,表明它能够适应不断变化的临床环境.
- SPC图表提供了一种有效的方法来监测模型性能和检测漂移.
- 在研究期间,不需要对BC管理工具进行重新校准或更正.
更多相关视频
04:09Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma
Published on: October 10, 2018
8.3K
10:28Dynamic Digital Biomarkers of Motor and Cognitive Function in Parkinson's Disease
Published on: July 24, 2019
15.2K
相关概念视频
01:30Pulse rhythm
817
Pulse rhythm refers to the pattern of pulsations within specific intervals, offering valuable insights into the regularity or irregularity of the heart's beats as observed through the pattern of pulsation within specific intervals. A regular pulse exhibits a consistent heart rate with uniform waveforms and pulsation force, variations of which can be classified as normal, weak, or bounding.
Conversely, an irregular pulse pattern is termed dysrhythmia, stemming from disruptions in cardiac...
Conversely, an irregular pulse pattern is termed dysrhythmia, stemming from disruptions in cardiac...
817
01:52Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K