ChatGPT for Sample-Size Calculation in Sports Medicine and Exercise Sciences: A Cautionary Note

Area of Science:

• Sport Sciences
• Sports Medicine
• Biostatistics

Background:

• Accurate sample size calculation is crucial for the validity and statistical power of research studies in sport sciences and sports medicine.
• Large language models (LLMs) like ChatGPT offer potential for assisting researchers, but their reliability for complex statistical tasks requires investigation.

Purpose of the Study:

• To evaluate the accuracy of ChatGPT in calculating sample sizes for sport sciences and sports medicine research.
• To identify potential errors and inconsistencies in LLM-generated sample size calculations.

Main Methods:

• Analysis of four published sport science/medicine research papers with varying study designs.
• Inputting all necessary statistical data (mean, SD, Z-values) and study design details into ChatGPT.
• Re-prompting ChatGPT with identical information to assess response reproducibility.

Main Results:

• ChatGPT correctly calculated the sample size for only one of the four studies (a randomized controlled trial).
• The LLM failed to correctly identify the appropriate formula for a survey paper's sample size calculation.
• Re-using the same prompt for one example yielded a different sample size calculation, indicating inconsistency.

Conclusions:

• Current LLMs like ChatGPT may produce errors and inconsistencies in sample size calculations, even with complete and accurate input.
• Researchers must exercise caution when using AI tools for statistical calculations and verify results.
• Future research should explore more advanced AI models and their capabilities in supporting scientific research tasks.