Related Experiment Video
Updated: Jul 2, 2026

Droplet Barcoding-Based Single Cell Transcriptomics of Adult Mammalian Tissues
Published on: January 10, 2019
Partial-label metric ceilings for evaluating gene regulatory networks inferred from single-cell foundation models
1University of Tuebingen, Computer Science Department, Germany.
None:
Gene regulatory network (GRN) benchmarks are typically interpreted as if curated references were complete, yet they are not. We formalize observed-metric ceilings under partial positive labels and reanalyze existing benchmark outputs across 15 methods and 5 references. The methods include six edge sets derived from a single-cell foundation model (scGPT) via attention and gradient attribution probes, evaluated alongside classical statistical and tree-based inferers and a random control. Under a missing-at-random (MAR) label model, we derive explicit ceilings for observed F1 and AUPR, propagate coverage uncertainty via Beta posteriors, and stress-test assumption violations. Crucially, because real curated references concentrate on extensively studied regulators, we replace stylized non-MAR tests with study-biased missingness grounded in two annotation databases: transcription factors assayed by ChIP-seq (CistromeDB/ChIP-Atlas) and genes carrying Reactome pathway annotations. Across 39 AUPR-evaluable rows, the best normalized F1 and AUPR ratios are 0.137 and 0.014, median normalized AUPR is 5.56×10-5, and only 3 of 39 rows exceed the observed random baseline; the foundation-model probes occupy the top of the ranking but still operate far below the observable ceiling. Restricting ground-truth edges to pathway-annotated genes leaves only ∼ 60% of edges observable and, at the empirically measured coupling between model scores and studied status, inflates observed AUPR by +0.040 above the MAR ceiling-intermediate between MAR (∼ 0) and the stylized worst case (+0.151). Missing labels therefore explain only part of the performance gap; substantial model-to-biology mismatch persists after ceiling correction, and evaluation claims should be narrowed accordingly.
Related Concept Videos
Cell Specific Gene Expression
Cell Specific Gene Expression
Constitutive and Regulated Gene Expression
Regulation of Expression at Multiple Steps
