Evaluating DNA Function Understanding in Genomic Language Models Using Evolutionarily Implausible Sequences

Area of Science:

• Synthetic Biology
• Computational Biology
• Genomics

Background:

• Genomic language models (gLMs) show potential for designing functional DNA sequences.
• A key challenge is distinguishing genuine functional understanding from pattern memorization in gLMs.
• Evaluating gLM generalization to novel, engineered sequences is crucial.

Purpose of the Study:

• Introduce Nullsettes, a novel framework to evaluate gLM prediction of loss-of-function (LOF) mutations.
• Assess whether gLMs understand sequence function or rely on training data patterns.
• Test gLM performance on synthetic expression cassettes without evolutionary history.

Main Methods:

• Developed the Nullsettes evaluation framework.
• Tested state-of-the-art gLMs on predicting LOF mutations in synthetic DNA.
• Analyzed model performance based on sequence likelihood and mutation impact.

Main Results:

• gLMs consistently failed to detect strong LOF mutations in synthetic constructs.
• Predictive accuracy decreased significantly when non-mutant sequences had lower model likelihood.
• Results suggest gLMs primarily match evolutionary patterns, not mechanistic gene expression.

Conclusions:

• Current gLMs exhibit significant limitations in generalizing to engineered genetic sequences.
• There is a critical need for evaluation methods that test for functional understanding.
• Future gLM development must prioritize mechanistic insights over pattern matching for reliable synthetic biology applications.