Applications of transformer-based language models in bioinformatics: a survey

Area of Science:

• Bioinformatics
• Computational Biology
• Natural Language Processing

Background:

• Transformer-based language models (e.g., BERT, GPT-3) show promise in bioinformatics due to similarities between biological sequences and natural language.
• These models offer remarkable interpretability and adaptability for biological data analysis.

Purpose of the Study:

• To review key developments in transformer-based language models for bioinformatics.
• To summarize their applications across various bioinformatics research areas.
• To identify challenges and opportunities for future research.

Main Methods:

• Detailed description of transformer model architecture.
• Summary of transformer applications in bioinformatics, including sequence analysis and drug discovery.
• Discussion of common challenges: data heterogeneity, computational cost, and interpretability.

Main Results:

• Transformers have demonstrated significant contributions to diverse bioinformatics tasks.
• Key challenges in applying transformers to bioinformatics include data issues and computational demands.
• Opportunities exist for novel bioinformatics applications leveraging transformer capabilities.

Conclusions:

• Transformer models offer powerful tools for advancing bioinformatics research.
• Addressing challenges in data, computation, and interpretability is crucial for wider adoption.
• Interdisciplinary collaboration can foster innovation in transformer-based bioinformatics.