Real-world evaluation of inter-consensus agreement on the Murad tool for appraising the methodological quality of...

The objective of this investigation was to evaluate the agreement rate on judgments made using the Murad tool by different systematic review teams.

Evaluating the methodological quality of case reports and case series is challenging but some tools do exist for this purpose. We leveraged the presence of studies that have been evaluated by different systematic review teams to assess the inter-consensus agreement among different teams on Murad tool domains.

Using a back-citation method, we identified systematic reviews that used the Murad tool and retrieved all the included primary studies. We selected studies that were assessed by more than 1 systematic review team. We calculated observed agreement and Gwet's agreement coefficient on judgments made about each signaling question.

We identified 982 systematic reviews that cited the Murad tool and collectively cited 59,080 references. The final data comprised of 81 duplicated case reports and series assessed by more than 1 systematic review team. Overall, the signaling questions had very high observed agreement with 5 of the 8 questions having agreement over 75%. The signaling questions with the highest agreement addressed the adequacy of the ascertainment of the exposure (coefficient 0.959), ascertainment of the outcome (coefficient 0.829), presence of a dose-response gradient (perfect agreement), and clarity of reporting (coefficient 0.755).

The current study demonstrates overall high agreement among different systematic review teams that used the Murad tool for appraisal of the same case series and case reports. Leveraging duplicated studies across systematic reviews is a feasible way to retrospectively assess the reliability of the tool.