Using leave-one-out cross validation (LOO) in a multilevel regression and poststratification (MRP) workflow: A...

Area of Science:

• Statistics
• Computational Statistics
• Survey Methodology

Background:

• Multilevel regression and poststratification (MRP) is increasingly used for population inference.
• Model validity in MRP is crucial but under-researched, particularly regarding validation techniques.
• Assessing the performance of different MRP models is essential for reliable population estimates.

Purpose of the Study:

• To evaluate the utility of leave-one-out cross-validation (LOO) for comparing Bayesian models in MRP.
• To investigate two approximate LOO calculations: Pareto smoothed importance sampling (PSIS-LOO) and a survey-weighted version (WTD-PSIS-LOO).
• To assess how accurately these LOO criteria rank models for both population-level and small-area estimation.

Main Methods:

• Utilized two simulation designs to test PSIS-LOO and WTD-PSIS-LOO performance.
• Examined model ranking accuracy for population estimands and small-area estimands.
• Applied the methods to real-world data from the National Health and Nutrition Examination Survey (NHANES).

Main Results:

• Neither PSIS-LOO nor WTD-PSIS-LOO consistently recovered the correct model order for population estimands, though they identified the best and worst models.
• Model performance varied across different small areas, complicating validation for small-area estimation.
• Model ranking showed slight improvement at smaller-area levels when considering different priors.
• Real-world NHANES data analysis corroborated simulation findings, indicating caution with PSIS-LOO for MRP.

Conclusions:

• PSIS-LOO-based model validation may not be fully suitable for evaluating MRP methods due to aggregation effects.
• The aggregation stage in MRP can obscure individual-level prediction errors, impacting LOO performance.
• These findings highlight the need for careful application and potential re-evaluation of LOO techniques in the context of MRP validation.