Enhancing cell type annotation for cancer transcriptomics using retrieval-augmented generation

Single-cell RNA sequencing has become an essential approach for investigating cellular heterogeneity in cancer, enabling detailed characterization of malignant cells, tumor microenvironments, and genetically distinct subpopulations. However, accurate and reproducible cell type annotation remains a major analytical bottleneck, limiting the reliability of downstream genetic and molecular interpretation. In this study, we develop a robust retrieval-augmented generation framework for automated cell type annotation that improves stability, biological grounding, and reproducibility. The proposed approach integrates hybrid lexical-semantic information retrieval with ontology-aware evaluation, combining keyword-based search and biomedical domain-specific embeddings to enhance semantic relevance. Retrieval performance is further strengthened through query expansion, paraphrasing, and lightweight reranking, while biological validity is ensured using Cell Ontology-grounded similarity scoring. The framework is systematically evaluated across nine publicly available single-cell RNA sequencing datasets spanning diverse human tissues. Compared with baseline prompting, naïve retrieval-based pipelines, and established annotation tools, the optimized framework demonstrates consistently higher annotation accuracy, reduced performance variability, and improved resolution of ambiguous and rare cell populations. By strengthening the reliability of cellular annotation, this approach facilitates more accurate analysis of genetic and molecular alterations relevant to cancer biology and provides a scalable foundation for translational and diagnostic applications in cancer genetics.