Cutting through the p-value: evaluating clinical relevance in surgical literature analyzing the approaches for...

The introduction of evidence-based medicine has challenged many concepts. In analyzing comparative study results, it is common to find narratives highlighting favorable outcomes based on a p-value of < 0.05, without understanding the clinical impact of the observed difference. Given the prevalence of this issue in hernia surgery research, we aimed to evaluate the prevalence of studies reporting a cutoff for clinical relevance in published comparisons of open, laparoscopic, and robotic inguinal hernia repair (IHR).

We searched Hernia, Surgical Endoscopy, Annals of Surgery, Surgery, World Journal of Surgery, and JAMA Surgery for articles comparing open, laparoscopic, and robotic IHR. Our search was performed according to the recent guidelines, comprising articles published since 2018. Articles analyzing non-clinical outcomes, such as cost-effectiveness, were excluded. Two authors independently screened the articles analyzing the presence of a clinical relevance cutoff definition of statistical significance, and if it suggested a superiority of a technique among others based solely on the p-value.

The initial search resulted in 62 articles, of which 8 were excluded, resulting in 54 included manuscripts. Among the included studies, 8 (14.8%) were randomized controlled trials (RCTs), while 46 (85.2%) were comparative cohort studies. Surprisingly, none of the studies defined a clinical relevance cutoff for the outcomes analyzed. Furthermore, only 6 (11.1%) studies highlighted that their findings may not be of clinical relevance. However, even among those 6 studies, 3 (50%) suggested a superiority of the approach based solely on the p-value, while the other 3 (50%) studies, despite finding a statistically significant difference, did not make this suggestion. 16 (29.6%) studies showed no statistically significant differences between the groups, but 2 (12.5%) of those still suggested a superiority of one of the surgical approaches. Among the RCTs, only 1 (12.5%) reported that their findings may not be of clinical relevance, while 1 (12.5%) suggested a benefit despite not finding statistically significant results.

Our study, encompassing the main journals in the surgical literature, demonstrated that the distinction between statistical and clinical relevance in hernia surgery, even in RCTs, remains inadequately addressed. There is a need for studies to define what difference in association measure is necessary to achieve clinical relevance for key outcomes in hernia surgery, such as recurrence, wound-related morbidity, and postoperative pain.