Distinguishing word identity and sequence context in DNA language models

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Transformer-based large language models (LLMs) show promise for analyzing biological sequences due to their natural language processing capabilities.
• Tokenization allows LLMs to learn complex relationships within biological sequences, similar to words in natural language.
• Masked token prediction during training enables LLMs to capture both local token identity and broader sequence context.

Purpose of the Study:

• To develop and apply methodologies for interrogating the learning processes of biological LLMs.
• To evaluate the interpretability and task-specific potential of LLMs trained on genomic data.
• To assess the capability of DNA language models to learn sequence context independent of tokenization strategies.

Main Methods:

• Utilized DNABERT, a DNA language model trained on the human genome using overlapping k-mers as tokens.
• Interrogated model predictions and extracted token embeddings to understand learned representations.
• Developed a novel fine-tuning task to predict subsequent tokens of varying lengths without overlap, isolating context learning.

Main Results:

• The DNABERT model with overlapping k-mers demonstrated limitations in learning extended sequence context.
• Learned embeddings primarily reflected token sequence identity rather than broader contextual information.
• Despite context learning challenges, the model achieved strong performance on genome-biology-specific fine-tuning tasks.

Conclusions:

• Overlapping k-mer tokenization in biological LLMs may hinder the learning of long-range sequence dependencies.
• LLMs with overlapping tokens are suitable for tasks where local token features are paramount and long-range context is less critical.
• There is a critical need for robust methods to interrogate and understand knowledge representation within biological LLMs.