scArchon: a scalable benchmarking framework for assessing single-cell perturbation models.

Area of Science:

• Computational Biology
• Genomics
• Bioinformatics

Background:

• Accurate prediction of cellular responses to perturbations like drug treatments is a key challenge in single-cell transcriptomics.
• Existing deep learning tools for this task lack systematic benchmarking across diverse datasets and metrics.

Purpose of the Study:

• To present scArchon, a reproducible and modular benchmarking platform for evaluating perturbation response prediction tools.
• To provide an unbiased and extensible framework for assessing single-cell RNA sequencing analysis methods.

Main Methods:

• Developed scArchon using Snakemake for reproducible and modular benchmarking.
• Compared leading perturbation prediction tools (scGen, CPA, trVAE, etc.) against baselines using six single-cell RNA-seq datasets.
• Assessed model performance using a combination of statistical and biological metrics, including gene-level evaluation.

Main Results:

• Demonstrated heterogeneous performance among evaluated tools, with trVAE, scGen, scPRAM, and scVIDR showing robust results across datasets.
• Identified instances where some tools underperformed compared to simple baselines.
• Highlighted that favorable quantitative scores do not always correlate with the retention of crucial biological perturbation signatures.

Conclusions:

• scArchon offers a unified and extensible foundation for standardized, large-scale benchmarking of perturbation prediction tools.
• The platform promotes methodological transparency and aims to accelerate development in single-cell perturbation modeling.
• Encourages the adoption of scArchon and sharing of containerized tools for advancing the field.