An Algorithm to Evaluate Methodological Rigor and Risk of Bias in Single-Case Studies

Area of Science:

• Methodology
• Research Design
• Evidence-Based Practice

Background:

• Critical appraisal scales are vital for assessing methodological rigor (MR) in intervention research, particularly for single-case designs (SCDs).
• Existing scales often lack differential weighting for internal validity, impacting accurate evidence strength assessment.
• Standardized classification of MR in SCDs is needed for reliable research evaluation.

Purpose of the Study:

• To develop and validate an algorithm based on the Risk of Bias in N-of-1 Trials (RoBiNT) Scale.
• To classify MR and risk of bias magnitude specifically for SCDs.
• To compare the algorithm's classifications with established standards like the What Works Clearinghouse (WWC).

Main Methods:

• An algorithm was derived from the RoBiNT Scale, incorporating differential weighting of internal validity items.
• The algorithm was applied to classify the MR of 46 SCD experiments.
• Classifications were compared against WWC standards for rigor in intervention studies.

Main Results:

• The algorithm classified 46 SCD experiments into categories ranging from Very High MR (4%) to Very Low MR (46%).
• Classification proportions showed strong agreement with WWC standards (28% met, 17% met with reservations, 54% did not meet).
• A strong association was observed between the RoBiNT-derived algorithm and WWC classifications.

Conclusions:

• The developed algorithm provides a robust method for classifying MR and risk of bias in SCDs.
• This approach offers a more nuanced assessment of evidence strength by considering internal validity weighting.
• The findings support the utility of the algorithm for consistent and reliable appraisal of SCD research.