Reproducibility of natural-language-driven statistical workflows compared with conventional software for diagnostic...

To evaluate whether a natural language-driven analytical workflow specified through a large language model (LLM) can accurately encode, execute, and reproducibly replicate a predefined statistical process traditionally performed in SPSS, and to assess its efficiency and auditability in urological research.

A dataset of 531 patients from a prostate cancer diagnostic study was analyzed in parallel using SPSS and ChatGPT's Data Analyst environment. Equivalent analytical procedures and variables were applied to calculate sensitivity, specificity, positive predictive value, negative predictive value, and diagnostic accuracy. Concordance between methods was assessed using absolute deviation and reproducibility rates. Processing time and user workload were also compared.

After two prompt-refinement steps to align analytical definitions, the LLM-based analysis achieved full concordance with SPSS across all diagnostic metrics, with a mean absolute deviation of <0.5%. Execution time for the stabilized artificial intelligence workflow was 8 minutes compared with 55 minutes for the conventional SPSS workflow, reflecting steady-state performance after workflow stabilization and excluding the initial prompt-refinement phase. Both methods produced identical diagnostic indices for clinically significant prostate cancer.

A natural language-driven workflow can accurately specify and reproducibly execute a predefined statistical analysis with full numerical concordance under controlled conditions, while substantially reducing execution time after workflow stabilization. These findings support LLM-based interfaces as complementary tools for reproducible and auditable analytical workflows, rather than replacements for conventional statistical engines.