Panero et al. (2016): Failure to replicate methods caused the failure to replicate results

Contrary to Kidd and Castano (2013), Panero et al. (2016) fail to find that reading literary fiction improves performance on an advanced test of theory of mind (ToM), the Reading the Mind in the Eyes Test. However, this commentary shows that the findings presented in Panero et al. (2016) are not reliable due to two striking threats to the internal validity of their studies that were not clearly disclosed or discussed in the manuscript or supplementary materials. First, no effective strategy was implemented to ensure that participants read their assigned texts, and examination of the data revealed many participants whose reading times indicate that they were not exposed to the manipulation. Second, further examination shows that two of the largest studies contributing to Panero et al. (2016) are not valid experiments due to a clear failure of random assignment to conditions. These threats to experimental internal validity make the conclusions presented in Panero et al. (2016) untenable. After removing cases in which participants were not exposed to the manipulation and the data from the two studies without random assignment, an analysis reveals that reading literary fiction improves ToM compared to reading popular genre fiction. This result is consistent with prior studies and indicates that a failure to carefully replicate the methods of Kidd and Castano (2013) led to the failure to replicate Kidd and Castano's (2013) results. (PsycINFO Database Record