Unsupervised evaluation of pre-trained DNA language model embeddings.

Area of Science:

• Genomics
• Computational Biology
• Machine Learning

Background:

• DNA Language Models (DLMs) show promise for genetics tasks but struggle with personal transcriptome variation.
• Current evaluation methods are computationally intensive and don't assess generalist learning capabilities.

Purpose of the Study:

• To introduce a computationally efficient framework for evaluating DLM embeddings.
• To assess the performance of six state-of-the-art DLMs using unsupervised metrics.
• To explore the correlation between unsupervised metrics and supervised task performance.

Main Methods:

• Proposed a framework using unsupervised numerical linear algebra-based metrics: RankMe, NESum, and StableRank.
• Generated embeddings from six DLMs (Nucleotide Transformer, DNA-BERT2, HyenaDNA, MistralDNA, GENA-LM, GROVER) on genomic datasets.
• Evaluated models using both unsupervised metrics and supervised classification tasks.

Main Results:

• DLM embeddings are high-dimensional and non-redundant, indicated by low correlations and variance.
• GENA-LM consistently performed strongly across unsupervised metrics and achieved high accuracy/F1 scores in supervised tasks.
• A positive correlation was observed between unsupervised metrics and supervised performance, validating unsupervised metrics as quality proxies.

Conclusions:

• Introduced a computationally efficient framework for DLM evaluation.
• GENA-LM, DNA-BERT2, and Nucleotide Transformer generally outperformed HyenaDNA and MistralDNA.
• Unsupervised metrics correlate with downstream performance, serving as effective proxies for DLM quality assessment.