DNA-aware evaluation and debiasing of sequence-to-function models | JoVE Visualize

Area of Science:

Genomics
Computational Biology
Bioinformatics

Background:

Sequence-to-function (S2F) models are crucial for interpreting functional genomics assays at base resolution.
Current S2F models are trained and evaluated using DNA-independent statistical methods, which may not fully capture the DNA-dependent nature of experimental assays.
A mismatch exists between DNA-independent model evaluation and DNA-dependent experimental measurements, necessitating DNA-aware evaluation approaches.

Purpose of the Study:

To investigate the DNA-dependency of experimental and S2F-predicted functional genomic tracks.
To develop and apply novel methods for evaluating S2F models that account for DNA sequence characteristics.
To identify and quantify the gap in DNA-decodability between experimental data and S2F model predictions.

Main Methods:

Utilized track-conditional genome language models (cgLMs) to probe DNA-dependency by predicting masked nucleotides.
Analyzed ATAC-seq and TF ChIP-seq data from GM12878 and K562 cell lines.
Developed DNA-dependency matching (DDM) objective and Critic-Guided Profile-Shape Editing (CGPSE) framework for post hoc debiasing.

Main Results:

A consistent masked DNA-decodability gap was observed between experimental tracks and most S2F-predicted tracks (e.g., BPNet, AlphaGenome).
S2F-predicted tracks showed higher accuracy and confidence in nucleotide recovery than experimental tracks, a gap not explained by standard profile-shape metrics.
ChromBPNet predictions were an exception, behaving closer to experimental data, and CGPSE partially reduced the gap for other models, albeit with a trade-off in profile-shape fidelity.

Conclusions:

Standard S2F model evaluation metrics do not fully capture DNA-dependent characteristics of functional genomics data.
A significant DNA-decodability gap exists between current S2F predictions and experimental measurements.
Novel DNA-aware evaluation and debiasing methods like DDM and CGPSE are necessary to improve the accuracy and biological relevance of S2F models.