机器学习模型的评估:信任和绩效
S Sousa1, S Paredes2,3, T Rocha1,4
1CISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290, Coimbra, Portugal.
Medical & biological engineering & computing
|June 7, 2024
概括
该研究开发了一种新的评估方法,同时评估机器学习模型的信任和性能. 基于规则的方法显示了对心血管风险评估的高度信任和卓越性能,有助于临床决策.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的机器学习
- 临床决策支持 临床决策支持
背景情况:
- 机器学习 (ML) 模型往往是黑子,阻碍了对医疗保健的信任和采用.
- 缺乏信任是ML在临床环境中广泛应用的一个重要障碍.
- 对于可靠的医疗保健ML工具,需要同时评估信任和绩效.
研究的目的:
- 开发和验证一个评估框架,评估ML模型的信任和性能.
- 将各种ML模型的信任度和性能与心血管风险分层临床标准进行比较.
- 根据信任和绩效指标,确定适合临床应用的ML模型.
主要方法:
- 信任评估包括模型的稳定性,信任区间 (95% CI) 和通过与临床证据的特征排名比较的可解释性.
- 使用几何平均值来评估性能.
- 使用葡萄牙心血管风险数据集 (N=1544) 进行了比较,比较了五种模型 (GRACE分数,逻辑回归,天真贝叶斯,决策树,基于规则的方法).
主要成果:
- 成功实施了对信任和绩效的同时评估.
- 基于规则的方法表现出高水平的运营信任.
- 基于规则的方法在表现方面超过了GRACE得分,并提高了医生接受度.
结论:
- 开发的评估方法有效地同时评估了ML模型的信任和性能.
- 基于规则的方法显示出在心血管风险评估中临床应用的巨大潜力.
- 提高ML模型的信任和性能可以提高医生的接受度,并帮助临床决策.
相关概念视频
01:24Confidence Coefficient
7.6K
The confidence coefficient is also known as the confidence level or degree of confidence. It is the percent expression for the probability, 1-α, that the confidence interval contains the true population parameter assuming that the confidence interval is obtained after sufficient unbiased sampling; for example, if the CL = 90%, then in 90 out of 100 samples the interval estimate will enclose the true population parameter. Here α is the area under the curve, distributed equally under...
7.6K
01:52Accuracy and Precision
8.8K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. Highly accurate...
8.8K
02:07Improving Translational Accuracy
10.1K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
10.1K
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
02:16Stereotype Content Model
14.7K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.7K
01:13Sensitivity, Specificity, and Predicted Value
284
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
284