Design-based fairness monitoring for healthcare prediction with complex survey data

Fairness evaluations for healthcare prediction models often use nationally representative survey data, but current practice rarely accounts for complex sampling when estimating uncertainty. We develop a design-based framework for fairness monitoring and use it to test temporal trends in healthcare expenditure prediction.

We analyzed nine annual waves (2015-2023) of the Medical Expenditure Panel Survey (MEPS), a nationally representative U.S. health survey with strata, primary sampling units, and person weights. Within each year, a Random Forest predicted high annual healthcare expenditure, defined as the top quartile among individuals with positive spending, using stratified five-fold cross-fitting so every eligible participant received an out-of-fold prediction. We evaluated demographic parity and equalized odds for sex, race/ethnicity, and poverty status. We used Taylor linearization to estimate design-based standard errors and 95% confidence intervals for survey-weighted fairness disparity ratios, then tested temporal trends with inverse-variance weighted least squares.

The race/ethnicity demographic parity ratio increased significantly (slope ＋0.132 per year, p=0.0007, 95% CI [0.077,0.186]), rising from 2.95 [2.73, 3.17] in 2015 to 4.00 [3.53, 4.47] in 2022. This increase is on the ratio (multiplicative) scale, driven by a steadily declining Hispanic flag rate, that is, growing relative under-flagging of Hispanic individuals. Against survey-weighted true high-expenditure rates, the model overstated the White/Hispanic gap by roughly 1.4-1.8 times, and this amplification itself increased significantly over the period (design-based log-scale trend p=0.004). Sex and poverty did not show significant trends. Sensitivity analyses with logistic regression, XGBoost, and a conventional held-out test split preserved the positive race/ethnicity trend.

The race/ethnicity demographic-parity ratio in high-expenditure prediction increased significantly across 2015-2023 while sex and poverty did not; this increase is on the multiplicative scale and reflects growing relative under-flagging of Hispanic individuals. The framework enables statistically valid longitudinal surveillance of healthcare AI using any complex survey with public design variables.