Why Is It so Hard to Do Good Science?

Area of Science:

• Cognitive Psychology
• Scientific Methodology
• Behavioral Economics

Background:

• Replication crisis in biomedical research highlights the need for improved objectivity.
• Psychological decision-making biases, well-documented in economics, also influence scientific judgment.
• Statistical methods alone cannot eliminate cognitive biases in scientific conclusions.

Purpose of the Study:

• To investigate the influence of cognitive biases on scientific judgment among scientists.
• To replicate classic decision-making experiments with a population of scientists.
• To identify inherent intuitions affecting scientific decision-making processes.

Main Methods:

• Replication of experiments by Kahneman and colleagues using written scenarios.
• Participants included scientists across various career stages.
• Analysis of scientists' written reactions to hypothetical research scenarios.

Main Results:

• Scientists exhibited similar cognitive biases as observed in earlier studies with undergraduates.
• Key biases include the drive for narrative coherence and pattern seeking in data.
• Tendency to overlook result likelihood irrespective of p-values was observed.

Conclusions:

• Inherent cognitive biases significantly impact scientific judgment and decision-making.
• Overcoming natural tendencies for quick decisions and narrative construction is essential for good science.
• Mitigating these biases requires learning and applying specific skills to enhance experimental rigor.