Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation

Area of Science:

• Functional genomics
• Systems biology
• Computational biology

Background:

• Predicting transcriptional responses to genetic perturbations is a key challenge in functional genomics.
• Existing computational methods aim to infer these responses but their generalizability is often overestimated.

Purpose of the Study:

• To evaluate the true predictive power of current methods for transcriptional responses to genetic perturbations.
• To introduce a new evaluation framework, Systema, to accurately assess prediction performance by focusing on perturbation-specific effects.

Main Methods:

• Quantified systematic variation (confounder-driven differences) across ten datasets from three technologies and five cell lines.
• Introduced Systema, an evaluation framework emphasizing perturbation-specific effects over systematic biases.
• Assessed the performance of existing prediction methods using Systema and standard metrics.

Main Results:

• Common evaluation metrics are susceptible to systematic variation, leading to overestimated predictive performance.
• Current methods struggle to generalize beyond systematic biases, failing to accurately predict responses to unseen perturbations.
• Systema framework revealed that predicting responses to novel perturbations is significantly more challenging than previously thought.

Conclusions:

• Existing methods' performance is inflated due to biases; true predictive power for unseen perturbations is lower.
• The Systema framework provides a more biologically meaningful assessment of perturbation response models.
• Disentangling systematic effects from true predictive performance is crucial for advancing perturbation response modeling in functional genomics.