Evaluating the Performance of R-Squared Measures in Multilevel Models

Multilevel Models (MLMs) have become a valuable tool in the behavioral and social sciences, providing a framework for analyzing clustered data structures commonly encountered in these fields. Unlike single-level regression, $R^2$ measures in MLMs become more intricate due to the need to account for sources of variance at different levels. Recently, Rights and Sterba (2019) introduced an integrative framework of MLM $R^2$ measures, providing a unifying approach to interpreting MLM $R^2$ measures in relation to specific substantive questions. While this framework represents a valuable resource for applied research, the $R^2$ measures have been defined in the population, and their performance across various conditions reflecting applied MLM practices remains unexplored. The present study evaluates the performance of the different MLM $R^2$ measures as estimators of their population values through Monte Carlo simulations. Among other factors, we examined how the number of level-1 and level-2 predictors, cross-level interactions, and random slopes affect the accuracy of the corresponding MLM $R^2$ measures. Results indicate that as the number of level-2 predictors increases, a greater number of clusters is required to ensure accurate estimates. The greater the number of level-1 predictors, cross-level interactions, and random slopes, increasing either the number of clusters or the number of observations per cluster leads to more accurate estimates.