Benchmarking Large Language Models for Predictive Modeling in Biomedical Research With a Focus on Reproductive Health | JoVE Visualize

Area of Science:

Computational biology
Bioinformatics
Artificial Intelligence in Genomics

Background:

Large language models (LLMs) are increasingly utilized in computational biology for automating data analysis code generation.
Standardized molecular datasets from Dialogue for Reverse Engineering Assessments and Methods (DREAM) Challenges in reproductive health were used.
Predictive modeling tasks included gestational age regression and preterm birth classification from gene expression, DNA methylation, and microbiome data.

Purpose of the Study:

To evaluate the capability of LLMs in generating functional R and Python code for predictive modeling in reproductive health omics data.
To assess LLM performance across diverse predictive tasks and identify top-performing models.
To compare the success rates of R versus Python code generation and analyze factors influencing performance.

Main Methods:

Eight different LLMs were prompted with task descriptions, data locations, and target outcomes for four predictive tasks.
LLM-generated R and Python code was executed to fit models, apply predictions, and generate graphics.
Performance was ranked based on task completion success and predictive accuracy on test datasets.

Main Results:

Four LLMs (o3-mini-high, 4o, DeepseekR1, Gemini 2.0) successfully generated error-free code for at least one task.
R code generation was significantly more successful (14/16 tasks) than Python (7/16), aided by Bioconductor packages.
OpenAI's o3-mini-high demonstrated superior performance, completing 7/8 tasks, with test set performance matching or exceeding original DREAM Challenge top teams.

Conclusions:

LLMs hold substantial potential to enhance exploratory data analysis and democratize predictive modeling in omics research.
Automating key components of analysis pipelines using LLMs can significantly increase research output, especially with standardized public datasets.
Further development of LLMs tailored for bioinformatics tasks could accelerate discoveries in reproductive health and other omics fields.