AnnQ: reference-based quantification of cellular abnormality at single-cell resolution

Reference-based annotation tools have become standard for cell type assignment in single-cell RNA sequencing, leveraging large-scale atlases to transfer labels to new datasets. However, a substantial fraction of cells often receive uncertain or ambiguous annotations-low confidence scores, competing label probabilities, or high entropy across cell types. These cells are typically treated as technical artifacts and filtered out, yet in perturbation experiments they may represent the biologically interesting deviations that investigators seek to identify. We present AnnQ (Annotation Quantification of cellular identity uncertainty), a Python framework that repurposes annotation uncertainty as a quantitative measure of cellular abnormality. AnnQ extracts uncertainty-aware features from probabilistic cell type assignments-including confidence, confidence gap, admixture ratio, and entropy-and computes an out-of-reference (OOR) score measuring each cell's deviation from a reference population in multivariate uncertainty space. Applying AnnQ to genetic perturbation and drug resistance datasets, we show that OOR scores detect aberrant cellular states that are not resolved by conventional clustering or differential abundance analyses. AnnQ provides a complementary approach for characterizing transitional and abnormal cell states at single-cell resolution. AnnQ is implemented in Python, and its source code and documentation are available on https://github.com/joonan-lab/AnnQ.git.