Methodological quality of systematic intervention reviews on vaccination

Systematic reviews (SRs) are pivotal in vaccination recommendation development. However, the reliability of their findings relies on methodological rigor. This study examines the methodological quality of vaccination-related SRs and aims to identify influencing factors.

We used the SYSVAC registry to draw 120 SRs on the efficacy or effectiveness of vaccination using a randomized block design. SRs published from 2011 to 2023 were considered. SR characteristics were extracted, and methodological quality was assessed by two reviewers using AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews 2). SRs were scored between 0 and 16 and categorized as high, moderate, low, or critically low quality. Mann-Whitney U test, chi-squared test, Fisher's exact test, and multivariable linear regression analysis were used to assess the influence of SR characteristics on AMSTAR 2 score. Secondary analysis of critically low-rated SRs was conducted to identify limitations in critical items.

Out of 120 SRs, 110 SRs were rated as of critically low-quality. The majority of critically low-rated SRs lacked the justification for excluding individual studies (n = 103, 93.6%) and protocol registration (n = 85, 77.3%). Median AMSTAR 2 score across all SRs was 10 (range 2-16). SRs published after 2017, Cochrane reviews, and SRs from authors with no conflicts of interest (CoI) had higher methodological quality than those published before 2017, non-Cochrane reviews, and SRs from authors with CoI (mean difference AMSTAR 2 score 2, 6, and 2, respectively; all p < 0.05). SRs published before 2017 had significant limitations in protocol registration, study selection in duplicate, and risk of bias assessment; and non-Cochrane reviews in protocol registration, justification for excluding individual studies, funding sources of studies, and impact of risk of bias assessment of individual studies (all p < 0.05). Management of conflicts was described only in a quarter of SRs with CoI (n = 9/43, 20.9%). The adjusted analysis showed that only publication year after 2017 and Cochrane review status had a significant independent relation on the AMSTAR 2 score (R² = 0.26; adjusted R² = 0.17).

The findings underscore the critical methodological shortcomings in vaccination-related SRs. Future efforts should prioritize adherence to established methodological standards and transparency, to enhance the impact of SRs in individual and decision-making.