Subjective evidence evaluation survey for many-analysts studies

Area of Science:

• Methodology
• Data Analysis
• Research Integrity

Background:

• Many-analysts studies assess empirical claims using multiple analysis teams on the same dataset.
• Current methods often rely on a single outcome metric (e.g., effect size) per team.
• This single metric limits a nuanced understanding of how analytical choices impact results.

Purpose of the Study:

• To develop a more comprehensive method for evaluating evidence in many-analysts studies.
• To assess the methodological appropriateness and strength of evidence for a hypothesis.
• To provide a richer understanding of the relationship between analysis choices and outcomes.

Main Methods:

• Utilized the Delphi consensus technique with 37 experts.
• Developed an 18-item subjective evidence evaluation survey (SEES).
• Applied the SEES to pilot data from a prior many-analysts study.

Main Results:

• The SEES provides a subjective evaluation of methodological appropriateness.
• The survey assesses the strength of evidence for a hypothesis from each analysis team.
• Pilot data demonstrated the SEES's utility in yielding richer evidence assessments.

Conclusions:

• The SEES offers a more nuanced approach to evidence evaluation in many-analysts studies.
• This method enhances the understanding of how analytical decisions influence research conclusions.
• The SEES contributes to improving research integrity and the interpretation of empirical claims.