Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for...

Area of Science:

• Social and Behavioural Sciences
• Scientific Methodology
• Research Integrity

Background:

• Replicating individual studies is resource-intensive, necessitating methods to predict research replicability.
• Existing techniques for predicting replicability may not meet accuracy standards.
• Ensuring the trustworthiness of scientific claims is paramount in research.

Purpose of the Study:

• To introduce and validate the repliCATS (Collaborative Assessments for Trustworthy Science) process for predicting research replicability.
• To provide a cost-effective method for evaluating scientific claims without full replication.
• To assess the scalability and potential applications of the repliCATS process.

Main Methods:

• The repliCATS process employs a structured expert elicitation approach, utilizing a modified Delphi technique.
• Experts evaluate research claims within the social and behavioural sciences.
• An online elicitation platform facilitates the assessment of a high volume of research claims.

Main Results:

• A validation study demonstrated the repliCATS process achieved 84% classification accuracy and an Area Under the Curve of 0.94.
• This accuracy meets or exceeds other known techniques for predicting replicability.
• The process is highly scalable, having assessed 3000 research claims over 18 months.

Conclusions:

• The repliCATS process offers a valid, accurate, and scalable method for predicting research replicability.
• It provides insights into the generalizability of scientific claims through qualitative data collection.
• Potential applications include alternative peer review and optimizing replication study allocation.