The Impact of Functional Form Complexity on Model Overfitting for Nonlinear Mixed-Effects Models

Nonlinear mixed-effects models (NLMEMs) allow researchers to model curvilinear patterns of growth, but there is ambiguity as to what functional form the data follow. Often, researchers fit multiple nonlinear functions to data and use model selection criteria to decide which functional form fits the data "best." Frequently used model selection criteria only account for the number of parameters in a model but overlook the complexity of intrinsically nonlinear functional forms. This can lead to overfitting and hinder the generalizability and reproducibility of results. The primary goal of this study was to evaluate the performance of eight model selection criteria via a Monte Carlo simulation study and assess under what conditions these criteria are sensitive to model overfitting as it relates to functional form complexity. Results highlighted criteria with the potential to capture overfitting for intrinsically nonlinear functional forms for NLMEMs. Information criteria and the stochastic information complexity criterion recovered the true model more often than the average or conditional concordance correlation. Results also suggest that the amount of residual variance and sample size have an impact on model selection for NLMEMs. Implications for future research and recommendations for application are also provided.