Beyond scores: A machine learning approach to comparing educational system effectiveness

Studies comparing large-scale assessment data among educational systems have been an important tool for understanding the differences in how education is delivered worldwide. Many of these studies do not go beyond reporting average student scores in a particular educational system. A more unbiased analysis would avoid the simple use of gross performance and consider educational system contexts. A common approach is to estimate effectiveness by the residuals of parametric linear models. These models rely upon strong assumptions regarding the data-generating process, and are limited to handling extensive datasets. To address this issue, our paper provides a new approach based on machine learning models. The new approach is flexible, allows paired comparison, and is model-independent. An analysis conducted in Brazil verifies the suitability of the method to explore differences in effectiveness between Brazilian educational administrative units at the regional and state levels from 2009 to 2019. Our results are consistent with the existing literature, but the methodology produced a number of new findings that were not observed in studies using more traditional approaches.