Fast leave-one-cluster-out cross-validation using clustered network information criterion.

Area of Science:

• Statistics
• Machine Learning
• Biostatistics

Background:

• Prediction models on clustered data require cluster-based validation for generalizability.
• Existing methods like Akaike information criterion (AIC) and Bayesian information criterion (BIC) may not adequately address cluster heterogeneity.
• Leave-one-cluster-out cross-validation is a robust but computationally intensive validation technique.

Purpose of the Study:

• Introduce a clustered estimator of the network information criterion (CNIC) as a fast approximation to leave-one-cluster-out deviance.
• Develop a method to assess model generalizability for prediction models with clustered data.
• Provide a more accurate model selection criterion for clustered data compared to AIC and BIC.

Main Methods:

• Derived a clustered network information criterion by modifying the standard network information criterion with a clustering-adjusted Fisher information matrix.
• Applied the CNIC to standard regression models with Gaussian or binomial responses for clustered data.
• Evaluated CNIC performance using simulation studies and an empirical example, comparing it to cluster-based cross-validation, AIC, and BIC.

Main Results:

• The clustered network information criterion (CNIC) provides a more accurate approximation to leave-one-cluster-out deviance than AIC and BIC.
• CNIC results in more accurate model size and variable selection, particularly when data exhibit strong clustering.
• CNIC imposes a greater penalty for stronger clustering, effectively preventing over-parameterization.

Conclusions:

• CNIC is a computationally efficient and accurate tool for model selection and validation in prediction models with clustered data.
• CNIC offers superior performance over traditional criteria like AIC and BIC when dealing with cluster heterogeneity.
• The proposed method enhances the reliability of prediction models developed on clustered datasets.