From articles to code: on-demand generation of core algorithms from scientific publications

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning

Background:

• Scientific software maintenance incurs high costs due to dependencies and bugs.
• Published papers are canonical records but translating them to code is difficult.
• Large language models (LLMs) offer potential for automated code generation from publications.

Purpose of the Study:

• To evaluate the feasibility of generating scientific code from published articles using LLMs.
• To assess LLM performance in reproducing established computational algorithms.
• To identify limitations in current publication standards for code generation.

Main Methods:

• Systematic evaluation of state-of-the-art LLMs.
• Tasking LLMs with implementing core algorithms using only scientific publications as input.
• Benchmarking across diverse algorithms including random forests, batch correction, and gene set enrichment analysis.

Main Results:

• LLMs successfully reproduced package-level functionality for many algorithms.
• LLM-generated code performance was often indistinguishable from established libraries.
• Discrepancies mainly stemmed from underspecified manuscript details, not LLM reasoning limitations.

Conclusions:

• Literature-driven code generation is feasible for well-specified algorithms.
• LLMs can reduce reliance on continuously maintained software libraries.
• Current publication standards need improvement to fully support automated code generation and reproducibility.