Related Experiment Video
Updated: Jun 16, 2026
05:47
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Evaluating large language models in biomedical data science challenges through a classroom experiment
Huifang Ma1, 1, Zhicheng Ji1
1Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27705.
Summary
Large language models (LLMs) show potential in designing machine learning solutions for data science challenges. Classroom experiments revealed LLMs can achieve competitive performance, even when used by nonexperts.
Area of Science:
- Computer Science
- Biomedical Informatics
- Machine Learning
Background:
- Large language models (LLMs) demonstrate strong algorithm design capabilities.
- Real-world effectiveness of LLMs in data science remains underexplored.
Purpose of the Study:
- To evaluate LLM performance in solving real-world biomedical data science challenges.
- To assess the impact of prompting strategies on LLM effectiveness.
Main Methods:
- Classroom experiment involving graduate students using LLMs on Kaggle.
- Focus on tabular data prediction tasks.
- Comparison of LLM-generated solutions against human participants.
Main Results:
- LLM submissions achieved prediction scores close to leading human participants.
- Gradient boosting methods were frequently recommended by LLMs and correlated with better performance.
- Self-refinement prompting strategy proved most effective, validated across multiple LLMs.
- LLM performance significantly decreased on tasks beyond tabular data prediction.
Conclusions:
- LLMs possess the potential to generate competitive machine learning solutions.
- LLMs can be valuable tools for data science tasks, even for nonexpert users.
- Further research is needed to optimize LLM performance for complex data science problems.
