Effect of tokenization on transformers for biological sequences.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Deep learning models are increasingly used in biological research, including bioinformatics and comparative genomics.
• Natural Language Processing (NLP) models have been applied to biological sequences, but differences in sequence structure pose challenges.
• Biological sequences are long and difficult to segment, unlike natural languages, hindering current machine learning models like transformers.

Purpose of the Study:

• To investigate the impact of alternative tokenization algorithms on various biological tasks.
• To assess improvements in accuracy and efficiency for deep learning models processing biological sequences.
• To explore the interpretability benefits of new tokenization strategies.

Main Methods:

• Studied eight diverse biological tasks, including protein function prediction, stability prediction, nucleotide sequence alignment, and protein family classification.
• Compared alternative tokenization algorithms against single-character tokenization.
• Trained tokenizers on a large dataset of over 400 billion amino acids.

Main Results:

• Alternative tokenization significantly increases accuracy and reduces input length compared to character-level tokenization.
• Trained tokenizers on large datasets decreased token count by over threefold.
• Database-specific tokenizers showed benefits for certain biological tasks.
• The tokenization approach allows for model interpretability by considering positional dependencies.

Conclusions:

• Tokenization is a critical component for future deep-network analysis of biological sequence data.
• Alternative tokenization strategies offer substantial improvements in efficiency and accuracy.
• The developed methods and resources are publicly available for further research.