Related Experiment Videos
Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical
Nikolai Ramadanov1,2, Hannah Jaeger3,4, Mikhail Salzmann3,4
1Center of Orthopaedics and Traumatology, Brandenburg Medical School, University Hospital Brandenburg, Brandenburg an der Havel, Germany. nikolai.ramadanov@gmail.com.
Background:
Large language models are increasingly integrated into biomedical research workflows, yet their ability to reliably reproduce complex statistical analyses remains insufficiently evaluated. While prior studies suggest that ChatGPT can replicate conventional meta-analyses, its performance in recalculating statistically demanding multilevel meta-analyses is unknown.
Methods:
We conducted a systematic review to identify published multilevel meta-analyses (i.e., models accounting for dependency among multiple effect sizes within studies) of hip-related clinical studies that provided extraction tables enabling independent recalculation. For each outcome, pooled estimates and heterogeneity parameters were recomputed using ChatGPT under predefined analytical assumptions and compared with statistician-derived results. Agreement was assessed using absolute and relative differences and Bland-Altman analyses for pooled effects, and qualitative comparison of heterogeneity measures (I², τ²) based on absolute differences.
Results:
Seven studies, each reporting a multilevel meta-analysis, comprising more than 40 pooled effect estimates were included. ChatGPT-derived pooled estimates showed close numerical agreement with human-derived results, with small median absolute differences for continuous (2.25) and binary outcomes (0.005) and minimal overall bias. Heterogeneity estimates demonstrated similar concordance, with predominantly small differences that did not alter qualitative interpretation. Larger deviations were limited to a small number of outcomes with extreme heterogeneity or inconsistent input structures.
Conclusion:
ChatGPT can reproducibly approximate the results of multilevel meta-analyses under predefined analytical conditions. These findings support the potential role of large language models as adjunct tools for reproducibility assessment and methodological validation in evidence synthesis, while emphasizing the continued need for expert oversight.
Related Concept Videos
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
Clinical Trials: Overview
Clinical Trials
There are four phases in a clinical trial. A phase one...
Bioavailability Study Design: Single Versus Multiple Dose Studies
Hazard Ratio
For example, in a clinical trial evaluating a...