Evaluating DNA Function Understanding in Genomic Language Models Using Evolutionarily Implausible Sequences

Genomic language models (gLMs) hold promise for generating novel, functional DNA sequences for synthetic biology. A critical challenge is determining whether gLMs understand sequence function or merely memorize training patterns derived from natural genomes. We introduce Nullsettes, an evaluation framework that measures how well models predict in silico loss-of-function (LOF) mutations in synthetic expression cassettes lacking evolutionary precedent. Across state-of-the-art gLMs, we find a consistent failure to detect strong LOF mutations. Predictive accuracy declines sharply when the original nonmutant has lower model likelihood, indicating reliance on evolutionary pattern-matching rather than mechanistic understanding of gene expression. These results expose core limitations in how gLMs generalize to engineered genetic constructs, and emphasize the need for evaluation and modeling strategies that explicitly test for functional understanding.