Multiple Confabulations Found in Bioinformatics Tasks Carried Out by Several Free Large Language Models.

Area of Science:

• Bioinformatics and computational genomics
• Application of Large Language Models (LLMs) in health sciences

Background:

• Bioinformatics heavily relies on computational analysis of experimental data.
• Evaluating the accuracy of general-purpose LLMs in scientific contexts is crucial.
• Confabulations (inaccurate answers) are a concern when using LLMs for research.

Purpose of the Study:

• To assess the performance of six freely available LLMs in common bioinformatics and computational genomics tasks.
• To identify the types and frequency of errors, specifically confabulations, generated by these LLMs.
• To explore the potential of LLMs in automating bioinformatics workflows.

Main Methods:

• Six LLMs (Gemini, ChatGPT, Grok, Claude, Llama, DeepSeek) were tested.
• Tasks included identifier conversion, DNA sequence simulation, polymorphism analysis, orthologue retrieval, gene ontology/pathway identification, volcano plot interpretation, and R code generation.
• Performance was evaluated based on accuracy and the presence of confabulations.

Main Results:

• A significant number of confabulations and errors were observed across various tasks for all tested LLMs.
• Performance varied depending on the task's complexity and the specific LLM used.
• Automatic generation of R code for data visualization emerged as a potentially reliable application.

Conclusions:

• Current general-purpose LLMs exhibit notable inaccuracies when applied to bioinformatics tasks.
• Automatic code generation is a promising area for LLM application in bioinformatics.
• Further investigation is required to understand and mitigate confabulations in LLMs for scientific research.