DNA-aware evaluation and debiasing of sequence-to-function models

Genome sequence-to-function (S2F) models are widely used to interpret base-resolution functional genomics assays. Most S2F models are trained and evaluated against observed counts and profile-shapes using statistical objectives and fidelity metrics. These choices are well motivated, but they are DNA-independent. At the same time, experimental measurements arise from DNA-dependent assays with distinct characteristics. This mismatch motivates a complementary DNA-aware evaluation of S2F-predicted and experimental functional genomic tracks.

We study DNA-dependency of experimental and S2F-predicted tracks using track-conditional genome language models (cgLMs). cgLMs predict masked nucleotides from a conditioning track under controlled DNA visibility. Across ATAC-seq and TF ChIP-seq peaks from GM12878 and K562, cgLM-probing reveals a consistent masked DNA-decodability gap between many experimental and S2F-predicted tracks. In particular, single-task (e.g. BPNet) and multi-task (e.g. AlphaGenome) S2F-predicted tracks enabled cgLMs to recover masked nucleotides with significantly higher accuracy and confidence than matched experimental tracks. Analyses of nonpeak and dinucleotide-shuffled sequences show that this gap is not confined to peaks and is not captured by standard DNA-agnostic profile-shape fidelity metrics alone. ChromBPNet Tn5-denoised predictions were an exception and behaved closer to the experimental regime, suggesting that staged training may reduce the gap. We then convert this diagnostic into a critic-derived objective, DNA-dependency matching (DDM), using a frozen multi-headed cgLM critic. We introduce Critic-Guided Profile-Shape Editing (CGPSE), a preliminary post hoc debiasing framework for frozen S2F models. In GM12878 ATAC-seq, CGPSE partially reduces the masked DNA-decodability gap for AlphaGenome and BPNet predictions, while exposing a tradeoff with profile-shape fidelity.

https://github.com/li-lab-mcgill/dna-aware-s2f-eval.