Discrepancies in Ratings of Behavioral Healthcare Interventions Among Evidence-Based Program Resources Websites

Area of Science:

• Behavioral Health
• Health Services Research
• Information Science

Background:

• Decision-makers in behavioral health need tools to identify evidence-based interventions.
• Evidence-based program resources websites (EBPRs) are such tools, but may disagree on intervention ratings.
• The reasons for these disagreements are not well understood.

Purpose of the Study:

• To examine how EBPRs rate interventions.
• To identify the sources of disagreement between EBPRs when rating the same intervention.
• To test the hypothesis that differing rating paradigms or evidence bases cause disagreements.

Main Methods:

• Identified 15 EBPRs for analysis.
• Coded EBPRs for 'tiers of evidence' and rating criteria used.
• Calculated Jaccard similarity for shared criteria and studies between co-rating EBPR pairs.
• Employed chi-square, correlation, and logistic regression analyses.

Main Results:

• A negative correlation was found between shared Cochrane Risk of Bias criteria and agreement on intervention ratings.
• No relationship existed between the number of shared studies and agreement on intervention ratings.
• Differences in rating criteria were the primary cause of disagreement between EBPRs.

Conclusions:

• Disagreements among EBPRs rating behavioral health interventions are mainly due to variations in their rating criteria.
• The specific studies used by EBPRs do not significantly influence agreement on intervention ratings.
• Clarifying EBPR methodologies is crucial for consistent intervention selection.