Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program...

Utilization of large-volume clinical registries for observational research has gained popularity in orthopaedic literature. However, concerns exist regarding inadequate reporting of methodology in this type of research. Despite these concerns, the reproducibility of such studies has not been adequately assessed in existing literature. This study aims to assess the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies on smoking as a risk factor for poor surgical outcomes by employing identical datasets and statistical methods.

A systematic PubMed search between 2013 and 2023 identified ACS-NSQIP studies involving hip or knee arthroplasty and smoking as a potential risk factor for poor surgical outcomes. Each study's methods were reproduced by a trained statistician based on the reported methodology. In cases where certain steps were not explicitly stated, the statistician made informed decisions to reproduce those steps. Adjusted odds ratios (aORs) and p-values (α = 0.05) were compared between the original and reanalyzed datasets.

The initial search yielded 43 studies, with 11 meeting inclusion criteria resulting in the reanalysis of 268 aORs. Upon reanalysis, 12.69% of the original studies' aORs changed in interpretation, while 13.43% experienced a change in statistical significance. The average magnitude change of each aOR across all studies was 17.22%, and the sample size (N) in reanalysis varied by up to 47.84%. Among the 11 commonly cited studies, approximately one in eight objective conclusions changed in interpretation or statistical significance.

Inconsistent reproducibility exists across many arthroplasty studies that utilize the ACS-NSQIP database. These findings highlight the importance of rigorous reporting of study methodology, data collection, and statistical analyses when utilizing large-volume databases in orthopaedic research. This burden of responsibility should be shared among authors, peer reviewers, and orthopaedic journals to confirm the accuracy and validity of published database research.

This study systematically reviewed and analyzed, in attempt to reproduce, published arthroplasty studies utilizing ACS-NSQIP database to assess smoking as a potential risk factor for poor surgical outcomes. All analyzed studies included Level III Evidence, therefore this current study compares reproduced Level III Evidence to the original Level III Evidence.