How persuasive is a good fit? A comment on theory testing

Area of Science:

• Philosophy of Science
• Psychology
• Quantitative Theory Evaluation

Background:

• Quantitative theories with free parameters are often accepted based on good data fits.
• This practice is problematic as it overlooks crucial information about theory flexibility and data variability.

Purpose of the Study:

• To critique the overreliance on good data fits for validating quantitative theories.
• To propose a more rigorous method for theory evaluation.

Main Methods:

• Conceptual analysis of theory validation in science and psychology.
• Examination of the limitations of using good fits as evidence.
• Proposal of an alternative framework for theory testing.

Main Results:

• Good data fits do not indicate a theory's flexibility or the data's constraining power.
• Historical and philosophical analysis reveals no significant progress from theories validated solely by good fits.
• A theory's predictive constraints and its ability to rule out alternative outcomes are more informative.

Conclusions:

• Theory validation should focus on predictive constraints and the ruling out of alternative outcomes, not just good data fits.
• A more robust approach to theory testing is necessary for scientific and psychological progress.
• Rethinking how we use data to support theories with free parameters is crucial.