L2G: Repurposing Language Models for Genomics Tasks.

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Foundation models (FMs) are transforming genomics, mirroring successes in natural language processing (NLP).
• Developing genomic FMs from scratch is computationally expensive and requires vast, high-quality datasets.
• Large language models (LLMs) in NLP benefit from industrial-scale data and infrastructure.

Purpose of the Study:

• To adapt existing LLMs for genomics, overcoming data and computational bottlenecks.
• To introduce L2G, a method for repurposing LLMs for diverse genomic applications.
• To evaluate the efficacy of LLM adaptation in genomics.

Main Methods:

• Leveraging cross-modal transfer from NLP transformers to genomic data.
• Employing neural architecture search (NAS) to adapt LLM architectures.
• Utilizing a novel three-stage training procedure for genomic tasks.

Main Results:

• L2G achieves superior performance on over half of tested genomics benchmark tasks.
• The model outperforms fine-tuned genomic FMs and task-specific models.
• L2G successfully identifies significant transcription factor motifs in enhancer activity prediction.

Conclusions:

• Pre-trained language models demonstrate remarkable generalizability to out-of-domain tasks like genomics.
• L2G offers an efficient, less resource-intensive approach to developing genomic models.
• This work opens new avenues for leveraging LLMs in genomic research.