Strong theory testing using the prior predictive and the data prior

In their seminal article, Roberts and Pashler (2000) highlighted that providing a good fit to empirical data does not necessarily provide strong support for a theory. For a good fit to be persuasive and for a theory to be strongly supported, the theory should have survived a strong test, in the sense that it is plausible that the theory might have failed the test. The most common way to accommodate the problem of the limited value of a good fit alone is to not only report a measure of goodness-of-fit, but also a measure of the complexity. A recent example of this line of reasoning is provided by Veksler, Myers, and Gluck (2015). In this article, I argue that whereas considering complexity provides useful information when theory testing, using complexity to gauge the severity of a test, or, equivalently, the persuasiveness of a good fit, is misguided. The reason is that complexity only provides information about the possibility of a bad fit, which does not guarantee a strong test. A condition for a test to be strong and a good fit to be persuasive is the demonstration of the plausibility of a bad fit. I provide a worked example of a more complete answer to assessing whether a good fit is persuasive. Providing a strong theory test requires the use of what can be called a data prior, which quantifies-before taking the empirical data into account-which outcomes are plausible. (PsycINFO Database Record (c) 2019 APA, all rights reserved).