性能:通过预测性和可解释性准备度公式评估模型性能
Leihong Wu1, Joshua Xu1, Weida Tong1
1Division of Bioinformatics and Biostatistics, National Center for Toxicological Research, FDA, Jefferson, AR, USA.
概括
我们开发了PERForm,这是一个统一的公式,将可解释性整合到用于AI模型评估的定量指标中. 这种方法提供了一个平衡的预测性和可解释性评估,增强AI模型选择和透明度.
科学领域:
- 人工智能的人工智能
- 计算毒理学计算毒理学
- 化学信息学 化学信息学
背景情况:
- 传统的人工智能模型评估通常将性能和可解释性分开,导致主观评估.
- 现有的定量指标主要集中在模型性能上,忽视了可解释性.
- 需要综合的量化措施来评估预测性和可解释性.
研究的目的:
- 介绍PERForm,这是一个统一的公式,用于在AI模型中量化测量预测性和可解释性.
- 为基于综合性能和可解释性评估和选择AI模型提供标准化的方法.
- 推进人工智能应用中的透明度和可解释性.
主要方法:
- 开发了PERForm公式,将可解释性作为权重因素纳入现有的统计绩效指标中.
- 在各种数据集 (DILIst,Tox21,MAQC-II) 和各种建模算法中应用了通用的PERForm公式.
- 评估了73个不同的终点,以证明公式的适用性和实用性.
主要成果:
- PERForm成功地将可解释性集成到定量AI模型评估中.
- 在数据集中展示了多样化的模型性能;AdaBoost在DILIst预测中表现出色,而线性回归对于大多数Tox21终点优越.
- 提供了模型性能和可解释性之间的权衡的定量证据.
结论:
- PERForm提供了一个强大的,定量框架来评估AI模型,平衡预测性和可解释性.
- 这种方法有助于更明智的模型选择,应用和开发.
- 这项研究有助于提高人工智能的透明度和可解释性.
相关概念视频
01:13Sensitivity, Specificity, and Predicted Value
305
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
305
01:19Variation
6.8K
An important characteristic of any set of data is the variation in the data. In some data sets, the data values are concentrated closely near the mean; in other data sets, the data values are more widely spread out from the mean. The most common measure of variation, or spread, is the standard deviation, which is the square root of variance.
When independent and dependent variables are plotted on a scatter plot, the slope of a line is a value that describes the rate of change between the two...
When independent and dependent variables are plotted on a scatter plot, the slope of a line is a value that describes the rate of change between the two...
6.8K
01:03Prediction Intervals
2.3K
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
2.3K
01:25Multiple Regression
3.0K
Multiple regression assesses a linear relationship between one response or dependent variable and two or more independent variables. It has many practical applications.
Farmers can use multiple regression to determine the crop yield based on more than one factor, such as water availability, fertilizer, soil properties, etc. Here, the crop yield is the response or dependent variable as it depends on the other independent variables. The analysis requires the construction of a scatter plot...
Farmers can use multiple regression to determine the crop yield based on more than one factor, such as water availability, fertilizer, soil properties, etc. Here, the crop yield is the response or dependent variable as it depends on the other independent variables. The analysis requires the construction of a scatter plot...
3.0K
01:16Goodness-of-Fit Test
3.3K
The goodness-of-fit test is a type of hypothesis test which determines whether the data "fits" a particular distribution. For example, one may suspect that some anonymous data may fit a binomial distribution. A chi-square test (meaning the distribution for the hypothesis test is chi-square) can be used to determine if there is a fit. The null and alternative hypotheses may be written in sentences or stated as equations or inequalities. The test statistic for a goodness-of-fit test is given as...
3.3K
01:15Receiver Operating Characteristic Plot
150
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
150