Evaluating large language models in biomedical data science challenges through a classroom experiment.

Area of Science:

• Computer Science
• Biomedical Informatics
• Machine Learning

Background:

• Large language models (LLMs) demonstrate strong algorithm design capabilities.
• Real-world effectiveness of LLMs in data science remains underexplored.

Purpose of the Study:

• To evaluate LLM performance in solving real-world biomedical data science challenges.
• To assess the impact of prompting strategies on LLM effectiveness.

Main Methods:

• Classroom experiment involving graduate students using LLMs on Kaggle.
• Focus on tabular data prediction tasks.
• Comparison of LLM-generated solutions against human participants.

Main Results:

• LLM submissions achieved prediction scores close to leading human participants.
• Gradient boosting methods were frequently recommended by LLMs and correlated with better performance.
• Self-refinement prompting strategy proved most effective, validated across multiple LLMs.
• LLM performance significantly decreased on tasks beyond tabular data prediction.

Conclusions:

• LLMs possess the potential to generate competitive machine learning solutions.
• LLMs can be valuable tools for data science tasks, even for nonexpert users.
• Further research is needed to optimize LLM performance for complex data science problems.