Potential and limitations of a large language model in the statistical review of comparative categorical data: an...

Statistical review is essential for research quality and integrity, yet traditional manual review is inefficient. Large language models (LLMs) offer potential support but are unreliable when used without guidance for precise calculations and raise concerns about accountability. This study evaluated whether a structured, rule-based prompt can reliably constrain an LLM to perform statistical review of comparative categorical data, and characterized both its feasibility and its inherent risks from an accountability perspective.

This study employed a two-stage design based on the DeepSeekV3.2. In the first stage, a structured prompt was developed through dozens of "test-fail-iterate" cycles using 20 published medical articles. The prompt assigned the LLM the role of a "statistics expert" and provided a closed set of computational rules and a "recognize data-select calculation formula-calculate" workflow for analyzing categorical data, including Pearson's Chi-square test, continuity correction, and McNemar's tests. In the second stage, the performance of the final prompt was evaluated on a test set of 20 independent manuscripts. The model's output was compared against the results calculated by a senior statistician (the gold standard). The primary outcome measures were the performance in statistical method selection and numerical computation, including accuracy, sensitivity (recall), specificity, positive predictive value, negative predictive value, F1 score, and Cohen's Kappa. Secondary measures included reproducibility and efficiency.

The test set consisted of 15 manuscripts with independent samples and 5 with paired samples. In the assessment of the appropriateness of statistical method selection for 148 analysis items, the model achieved an accuracy of 99.3% (147/148), a sensitivity of 96.2% (25/26) (F1=98.0%, κ=0.976). For the test of computational consistency in 97 independent sample tests, the accuracy for χ² value consistency was 94.8% (92/97) (F1=89.3%, κ=0.859), and for P-value consistency, it was 96.9% (94/97) (F1=90.9%, κ=0.891). In the paired-sample analysis, the model's methods and results were in perfect agreement with the manual review, and prompt optimization eliminated discrepancies in degrees-of-freedom calculation rules. Efficiency analysis showed no statistically significant difference in time consumption between the model (407 s) and manual review (374 s) (P=0.601). In reproducibility tests, the intraclass correlation coefficients for both χ² values and P-values exceeded 0.91. However, qualitative analysis revealed 3 typical failure modes in the task workflow: (1) Instability: The model's failure to produce identical outputs across repeated runs, manifesting as inconsistent data extraction or the failure to process all designated tasks (scope neglect). (2) Performance degradation/"lazy" behavior: A decline in execution quality on long or complex tasks, often characterized by the model abandoning its reasoning process to copy author-provided values without verification. (3) Anchoring effect: The model's tendency to over-rely on author-provided statistical values (the "anchor"), causing its verification process to be unduly influenced.

A structured, rule-based prompt can guide the DeepSeek to achieve high accuracy in standardized statistical review tasks, but its reliability is contingent on operational stability. Inherent failure modes, including performance instability and a strong anchoring effect on author-provided data, persist and can lead to significant errors, particularly when source data are flawed. These findings suggest that the the DeepSeek is not suitable for autonomous auditing. Their most appropriate application is as assistive tools within a human-in-the-loop framework, where rigorous human supervision is essential for risk mitigation and to maintain ultimate accountability.