Interpreting embeddings from genome and protein language models

Following advances in the field of natural language processing related to the transformer and the attention mechanism, many large language models have been trained using DNA and protein databases. While these models are designed for next-token prediction with the goal of generating novel sequences, researchers have found that the intermediate representations, or embeddings, of the DNA and amino acid sequences as they pass through these models can be mined for biological insights. Recently, more emphasis has been placed on studying the models' embeddings than model outputs. Genome language model embeddings have been used to predict the severity of single nucleotide variants, intron/exon boundaries, and anti-CRISPR activity. Protein language model embeddings have been used for predicting remote homology, protein structure, and protein function. These findings demonstrate that model embeddings provide a high-resolution, alignment-free framework for understanding the genome and proteome, and offer a transformative approach to enzyme engineering and functional genomics.