Evaluating large-scale propensity score adjustment when sample size is small

Area of Science:

• Epidemiology
• Biostatistics
• Health Data Science

Background:

• Large-scale propensity score (LSPS) models are vital for controlling confounding in observational research.
• Concerns exist regarding the reliability of LSPS in small-sample settings due to potential issues with accurate propensity score estimation and covariate balance.
• Small datasets are often excluded, losing valuable information.

Purpose of the Study:

• To evaluate the performance of LSPS models in small-sample settings.
• To assess the balancing and bias reduction capabilities of LSPS models under varying sample sizes.
• To compare local LSPS models within data subsets to a global LSPS model trained on the full dataset.

Main Methods:

• LSPS models were evaluated using six target-comparator pairs across partitioned real-world data sources.
• Performance was assessed based on covariate balance, treatment effect bias, and precision after empirical calibration.
• Methods included real negative controls and synthetic positive controls, emulating a distributed data network analysis.

Main Results:

• LSPS models significantly reduced bias and improved precision with empirical calibration.
• Propensity score (PS) matching demonstrated greater resilience to small sample sizes than PS stratification.
• Standard balance diagnostics were unreliable below 20,000 participants, unlike a chance imbalance diagnostic.

Conclusions:

• LSPS models offer reliable bias reduction in small-sample settings, suitable for federated analyses.
• Standard balance diagnostics can be misleading in small samples; alternatives like significance checking are recommended.
• When LSPS models inadequately reduce bias, supplementary adjustment strategies are necessary.