Nucleotide Transformer: building and evaluating robust foundation models for human genomics.

Area of Science:

• Genomics
• Bioinformatics
• Machine Learning

Background:

• Predicting molecular phenotypes from DNA sequences is a significant challenge in genomics.
• Limited annotated data and task-specific learning hinder progress.

Purpose of the Study:

• To develop and evaluate large-scale foundation models for DNA sequence analysis.
• To demonstrate the effectiveness of these models in various genomics applications, especially in low-data scenarios.

Main Methods:

• Pre-trained foundation models, named Nucleotide Transformer, with parameters ranging from 50 million to 2.5 billion.
• Integration of data from 3,202 human genomes and 850 diverse species genomes.
• Fine-tuning models for specific genomics tasks, including genetic variant prioritization.

Main Results:

• Nucleotide Transformer models generate context-specific nucleotide sequence representations.
• Accurate predictions are achieved even in low-data settings.
• Models learn to focus on key genomic elements without supervision, enhancing genetic variant prioritization.

Conclusions:

• Foundation models offer a widely applicable approach for accurate molecular phenotype prediction from DNA.
• The developed models can be fine-tuned cost-effectively for diverse genomics applications.
• This work advances the use of large-scale pre-trained models in genomic research.