Improving real-world data analysis: Revisiting claims that gender meaningfully impacts marathon pacing

The use of real-world data in sports is increasing but without a commensurate increase in analytical rigour or interpretation. This study reexamines a seminal analysis which claims to show gender differences in marathon pacing are 'robust'. The earlier work suffers from serious methodological flaws common to real-world data analysis: (1) lack of quality control in the raw data, (2) overuse and misuse of ratio-based metrics, (3) failure to validate the assumptions of statistical models. A proper and valid analysis of the exact same real-world data indicates that there are some potentially interesting differences in pacing between genders, but the differences are only evident in younger and slower runners. While the previous, invalid analysis has been used to suggest there is some sort of biological mechanism for a gender difference in marathon pacing, our valid analysis is more suggestive of a social difference in this large cohort of recreational marathon runners. This case highlights the need for researchers to prioritize the quality, not just the quantity, of real-world data analyses. Moreover, researchers should avoid defaulting to sensational interpretations when more cautious, grounded explanations may be equally - if not more - plausible.