ChatGPT's performance in sample size estimation: a preliminary study on the capabilities of artificial intelligence

Area of Science:

• Clinical Research Methodology
• Statistical Computing
• Artificial Intelligence in Healthcare

Background:

• Artificial intelligence (AI) and large language models (LLMs) like ChatGPT are increasingly used in clinical research.
• The utility of these AI tools for specialized statistical tasks, such as sample size estimation, is not well-understood.

Purpose of the Study:

• To evaluate the accuracy and reproducibility of ChatGPT-4.0 and ChatGPT-4o in performing sample size estimations.
• To compare the performance of different versions of ChatGPT for statistical calculations in research.

Main Methods:

• Sample size calculations were performed for 24 standard statistical scenarios using ChatGPT-4.0 and ChatGPT-4o.
• Accuracy was measured by absolute percentage error against reference values.
• Reproducibility was assessed by comparing results from independent chat sessions.

Main Results:

• Both ChatGPT-4.0 and ChatGPT-4o provided reasonably accurate sample size estimates, with most errors below 5%.
• ChatGPT-4o demonstrated superior accuracy and slightly better reproducibility compared to ChatGPT-4.0.
• Observed inconsistencies highlight the need for careful review of AI-generated estimates.

Conclusions:

• ChatGPT-4.0 and ChatGPT-4o can be useful tools for preliminary sample size estimation in standard research scenarios.
• Users must exercise caution and seek expert validation for AI-generated statistical results.
• Further investigation is needed for complex statistical tasks and a wider array of AI models.