Related Experiment Video
Updated: Jan 5, 2026

Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Impact of choice of quality appraisal tool for systematic reviews in overviews
Dawid Pieper1, Tim Mathes, Michaela Eikermann
1Institute for Research in Operative Medicine, Witten/Herdecke University, D- 51109 Cologne, Germany.
Objective:
The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews).
Methods:
Based on a published overview investigating the hospital volume-outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five-point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated.
Results:
The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter-rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers.
Conclusion:
The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items.
More Related Videos
Related Concept Videos
Bioavailability Study Design: Absolute Versus Relative Bioavailability
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Quality Assurance
Bioequivalence: Overview
Decision Making: P-value Method
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
Pharmacokinetic Models: Comparison and Selection Criterion
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.

