Out-of-the-box bioinformatics capabilities of large language models (LLMs)

Area of Science:

• Bioinformatics and Computational Biology
• Artificial Intelligence in Life Sciences
• Genomic Data Analysis

Background:

• Bioinformatics, the analysis of biological sequences (DNA, RNA, protein), is crucial for biological research.
• The computational nature of bioinformatics makes it highly suitable for AI-driven automation and acceleration of scientific discovery.
• General-purpose Large Language Models (LLMs) are increasingly explored for their potential across scientific disciplines.

Purpose of the Study:

• To assess the bioinformatics capabilities of popular general-purpose LLMs.
• To compare LLM performance against human performance on a standardized set of bioinformatics questions.
• To identify strengths and weaknesses of LLMs in distinct bioinformatics domains.

Main Methods:

• Utilized 104 questions from Rosalind, an educational bioinformatics platform.
• Evaluated three LLMs: GPT-3.5, Llama-3-70B, and GPT-4o.
• Compared LLM performance to data from 110 to 68,760 human participants globally.

Main Results:

• GPT-3.5 achieved the highest accuracy (58%), followed by Llama-3-70B (47%) and GPT-4o (47%).
• 71% of questions were answered correctly by at least one LLM.
• LLM performance varied by category, excelling in DNA analysis but struggling with sequence alignment and genome assembly, mirroring human performance trends.

Conclusions:

• LLMs demonstrate capacity for biological knowledge, reasoning, statistical analysis, and code generation in bioinformatics.
• LLM performance in bioinformatics tasks often correlates with human performance, indicating potential as tools for scientific inquiry.
• Further development of LLMs is warranted to enhance their utility and reliability in complex bioinformatics applications.