How Consistently Do 13 Clearinghouses Identify Social and Behavioral Development Programs as "Evidence-Based"?

Area of Science:

• Social and behavioral sciences
• Program evaluation
• Evidence-based practice

Background:

• Clearinghouses assess social and behavioral development programs using scientific criteria to determine evidence-based status.
• These verdicts are published online for stakeholders, including policymakers and the public.

Purpose of the Study:

• To compare the criteria and preferences of 13 clearinghouses in evaluating social and behavioral programs.
• To assess the consistency of program rankings across different clearinghouses.
• To investigate the reasons behind discrepancies in clearinghouse conclusions through case studies.

Main Methods:

• Comparative analysis of causal design and analysis preferences across 13 clearinghouses.
• Statistical estimation of inter-rater reliability in program effectiveness rankings.
• Qualitative case study analysis to explore reasons for differing conclusions.

Main Results:

• Most clearinghouses prioritize randomized controlled trials but vary in valuing implementation data and quasi-experimental designs.
• A significant majority of sampled programs (82%) were reviewed by only one clearinghouse.
• Agreement on program effectiveness was found for only 55% of programs reviewed by multiple clearinghouses, with higher consensus on ineffectiveness.

Conclusions:

• Discrepancies in clearinghouse conclusions stem primarily from differing requirements for independent replication and temporal persistence of effects.
• Lack of consensus on key methodological standards hinders the consistent application of "evidence-based" in social and behavioral sciences.
• Establishing shared scientific consensus on evaluation criteria is crucial for advancing evidence-based practice.