A Complete SMOCkery: Daily Online Testing Did Not Boost College Performance

Area of Science:

• Educational Technology
• Higher Education Research
• Academic Integrity

Background:

• A widely cited 2013 study by Pennebaker et al. reported significant student performance gains and reduced achievement gaps using a daily online testing system.
• This study led to substantial investment in university projects and the development of numerous synchronous massive online courses (SMOCs).

Purpose of the Study:

• To critically re-evaluate the data and methodology of the Pennebaker et al. (2013) study.
• To identify potential threats to internal validity that may have been overlooked in the original research.
• To provide a more accurate interpretation of the findings regarding online learning interventions.

Main Methods:

• Secondary data analysis of the Pennebaker et al. (2013) study dataset.
• Investigating potential confounding factors such as selection bias and differential attrition.
• Assessing the internal validity of the original study's conclusions.

Main Results:

• The reported improvements in student performance in concurrent courses can be fully attributed to selection bias.
• Reductions in achievement gaps are likely explained by differential attrition rates among student groups.
• The original study inadequately addressed critical threats to internal validity.

Conclusions:

• The claims of the Pennebaker et al. (2013) study regarding the efficacy of daily online testing are not supported by the data when controlling for methodological limitations.
• Findings suggest that the success of synchronous massive online courses (SMOCs) may be overestimated due to unaddressed biases.
• There is a critical need for rigorous scrutiny of research, particularly in open-access or pay-to-publish journals, to prevent the propagation of unsupported claims in educational technology.