The shaky foundations of simulating single-cell RNA sequencing data.

Area of Science:

• Computational Biology
• Genomics
• Bioinformatics

Background:

• Hundreds of single-cell RNA sequencing (scRNA-seq) datasets are emerging.
• A growing number of computational tools require robust benchmarking.
• Benchmark studies often rely on simulated data for method evaluation.

Purpose of the Study:

• Evaluate synthetic scRNA-seq data generation methods.
• Assess the ability of simulators to mimic experimental data.
• Investigate the impact of simulators on downstream analysis comparisons.

Main Methods:

• Evaluated synthetic scRNA-seq data generation methods.
• Compared gene- and cell-level quality control summaries.
• Quantified batch- and cluster-level effects.
• Investigated simulator effects on clustering and batch correction comparisons.

Main Results:

• Most simulators introduce artificial effects in complex designs.
• Simulators yield over-optimistic performance for data integration.
• Simulator-based comparisons may provide unreliable rankings for clustering methods.

Conclusions:

• Current synthetic data simulators have limitations in mimicking real scRNA-seq data.
• The reliability of benchmarking computational tools using simulated data is questionable.
• Further research is needed to identify effective quality control summaries for simulation-based comparisons.