StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis.

Area of Science:

• Machine Learning
• Data Science
• Statistical Analysis

Background:

• Large language models (LLMs) show promise for automating statistical analysis.
• Assessing the accuracy of LLM-generated code is essential before widespread adoption.
• A lack of benchmark datasets hinders the evaluation of LLMs in statistical coding.

Purpose of the Study:

• Introduce StatLLM, an open-source dataset for evaluating LLM performance in statistical analysis.
• Provide a comprehensive benchmark for assessing the accuracy and quality of LLM-generated statistical code.
• Facilitate advancements in natural language processing metrics, LLM capabilities, and statistical software development.

Main Methods:

• Developed StatLLM dataset with statistical analysis tasks, LLM-generated SAS code (GPT-3.5, GPT-4, Llama-3.1 70B), and human evaluation scores.
• Included diverse statistical analysis tasks with problem descriptions, dataset details, and human-verified SAS code.
• Collected expert human evaluations on correctness, effectiveness, readability, executability, and output accuracy of generated code.

Main Results:

• The StatLLM dataset provides a structured framework for evaluating LLM-generated statistical code.
• Human evaluations offer insights into the strengths and weaknesses of different LLMs in statistical coding.
• The dataset enables quantitative assessment of LLM performance across various statistical tasks.

Conclusions:

• StatLLM is a valuable resource for benchmarking and improving LLMs in statistical analysis.
• The dataset supports the development of more accurate and reliable AI tools for data science and machine learning.
• Future work can leverage StatLLM to enhance NLP metrics and next-generation statistical software.