A comprehensive power-analytic investigation of research in medical education

A total of 230 major articles in volumes 55 through 57 of the Journal of Medical Education were reviewed for a comprehensive power-analytic investigation of research in medical education. Three statistical power determinations were made for each of 2,220 reported tests of significance, and the average power for detecting a range of possible treatment effects was calculated for each of the 100 studies subsequently included in the analysis. Among other findings, fully 91 percent of the 100 articles analyzed had less than a 50-50 chance of detecting a "small" treatment effect. Average power figures from similar surveys in other disciplines demonstrate that the problem of low statistical power is not unique to research in medical education. Additionally, the practical consequences of low statistical power are outlined, and workable guidelines for reporting the information necessary for the independent evaluation of published studies are provided.