Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type
Related Experiment Video
Updated: Nov 7, 2026
Discrimination and Characterization of Heterocellular Populations Using Quantitative Imaging Techniques
Published on: June 30, 2017
Motivation:
Single-cell cell-type classifiers are often evaluated using random cell-level splits, although cells from the same donor are not independent. We asked how evaluation design affects estimates of cross-donor generalization.
Results:
Across PBMC and lung datasets, random cell-level splitting consistently yielded higher macro F1 than donor-held-out evaluation across multiple expression representations. This gap persisted after batch covariate adjustment and donor-level mixed-effects modeling. In contrast, evaluation gaps were minimal in near-ceiling kidney, pancreas, and blood benchmarks and in a randomized-label floor control. These results show that evaluation bias depends on the underlying classification regime and that donor-held-out evaluation is necessary when claims concern generalization to unseen individuals.
Availability And Implementation:
Code and analyses are available on GitHub at https://github.com/danielchen05/scRNA-cross-donor-generalization/ and archived at Zenodo at 10.5281/zenodo.23669184. All data are publicly available and cited.
Contact:
ajbattle@jhu.edu.
