Trialling Meta-Research in Comparative Cognition: Claims and Statistical Inference in Animal Physical Cognition

Area of Science:

• Comparative cognition
• Animal behavior
• Scientific methodology

Background:

• Replicability and statistical inference are critical concerns in scientific disciplines, including animal cognition research.
• Assessing the methods and claims in animal physical cognition studies is essential for understanding research practices.

Purpose of the Study:

• To evaluate how researchers formulate and publish claims regarding animal physical cognition.
• To examine the statistical inferences employed to support these claims in published research.

Main Methods:

• Surveyed 116 experiments from 63 papers on animal physical cognition, covering 43 species.
• Analyzed common tasks such as trap-tube, tool use, means-end understanding, string-pulling, object choice, object permanence, and access tasks.
• Examined statistical claims, sample sizes (median of 7), and p-value distributions.

Main Results:

• Between 44% and 59% of papers made positive claims, 24%-46% inconclusive, and 10%-17% negative claims.
• Negative results (failures to pass tasks) were published, indicating some reporting of non-significant findings.
• Evidence suggests publication bias, with a drop in p-values above .05, and potential overestimation of effect sizes at the group-level.

Conclusions:

• While researchers appear to make correct individual-level inferences, group-level evidence strength is weaker.
• Publication bias is present, and effect sizes may be overestimated, necessitating careful interpretation of comparative cognition findings.
• Future research should prioritize developing valid and reliable measurement tools in animal cognition studies.