Comparative assessment of large language models for microbial phenotype assignment

Area of Science:

• Microbiology
• Bioinformatics
• Artificial Intelligence

Background:

• Large language models (LLMs) are increasingly used for knowledge extraction from text.
• Their reliability and coverage in biological data, especially for microbial phenotypes, are not well understood.
• Microbial phenotypes are crucial for understanding microbial characteristics, roles, and applications, yet data remain sparse for many species.

Purpose of the Study:

• To systematically assess the biological knowledge encoded in publicly available LLMs for microbial species phenotype assignment.
• To evaluate the performance of various LLMs, including state-of-the-art models, in this task.
• To determine the utility and limitations of text-based LLMs for microbiology phenotype characterization.

Main Methods:

• Systematic evaluation of up to 57 LLMs.
• Testing performance across diverse microbial phenotypes.
• Analysis of model self-reported confidence in relation to accuracy.

Main Results:

• LLMs achieved accurate phenotype assignments for numerous microbial species.
• Performance varied significantly across different models and specific traits.
• No single LLM model consistently outperformed others across all evaluations.
• Model confidence ratings correlated with accuracy, enabling prioritization of phenotype assignments.

Conclusions:

• Text-based LLMs offer utility for microbial phenotype characterization.
• Significant limitations and variability in performance exist among current LLMs.
• Model confidence can serve as a useful metric to guide the application of LLM-derived phenotype data.