AUC-PR is a More Informative Metric for Assessing the Biological Relevance of In Silico Cellular Perturbation...

Area of Science:

• Computational biology
• Systems biology
• Genomics

Background:

• In silico perturbation models predict cellular responses to genetic or drug-induced changes, aiming to reduce in vitro experiments.
• Current evaluation metrics like R-squared assess overall accuracy but overlook biologically critical outcomes such as identifying differentially expressed genes (DEGs).

Purpose of the Study:

• To introduce and validate a novel evaluation framework using the Area Under the Precision-Recall Curve (AUC-PR) metric for assessing DEG prediction performance.
• To systematically benchmark computational models for in silico perturbations using both single-cell and pseudo-bulk datasets.

Main Methods:

• Developed an evaluation framework incorporating the AUC-PR metric to quantify the precision and recall of DEG predictions.
• Applied the framework to benchmark various computational models on single-cell and pseudo-bulk perturbation datasets.

Main Results:

• A significant discrepancy was observed between R-squared and AUC-PR performance metrics.
• Models with high R-squared values demonstrated poor performance in accurately identifying DEGs, indicated by low AUC-PR values.
• The proposed AUC-PR framework provides a more biologically relevant assessment of model capabilities.

Conclusions:

• Traditional evaluation metrics like R-squared are insufficient for assessing the biological relevance of in silico perturbation models.
• The AUC-PR metric offers a superior evaluation of a model's ability to predict differentially expressed genes.
• This work advances the reliable application of computational models in cellular perturbation research by emphasizing biologically meaningful performance.