Large inherent variability in data derived from highly standardised cell culture experiments

Cancer drug development is hindered by high clinical attrition rates, which are blamed on weak predictive power by preclinical models and limited replicability of preclinical findings. However, the technically feasible level of replicability remains unknown. To fill this gap, we conducted an analysis of data from the NCI60 cancer cell line screen (2.8 million compound/cell line experiments), which is to our knowledge the largest depository of experiments that have been repeatedly performed over decades. The findings revealed profound intra-laboratory data variability, although all experiments were executed following highly standardised protocols that avoid all known confounders of data quality. All compound/ cell line combinations with > 100 independent biological replicates displayed maximum GI50 (50% growth inhibition) fold changes (highest/ lowest GI50) > 5% and 70.5% displayed maximum fold changes > 1000. The highest maximum fold change was 3.16 × 10¹⁰ (lowest GI50: 7.93 ×10^-10 µM, highest GI50: 25.0 µM). FDA-approved drugs and experimental agents displayed similar variation. Variability remained high after outlier removal, when only considering experiments that tested drugs at the same concentration range, and when only considering NCI60-provided quality-controlled data. In conclusion, high variability is an intrinsic feature of anti-cancer drug testing, even among standardised experiments in a world-leading research environment. Awareness of this inherent variability will support realistic data interpretation and inspire research to improve data robustness. Further research will have to show whether the inclusion of a wider variety of model systems, such as animal and/ or patient-derived models, may improve data robustness.