A tiered spatial validation workflow for geographic prediction models

Area of Science:

• Geographic machine learning
• Spatial statistics
• Geospatial data science

Background:

• Standard validation methods assume data exchangeability, which is violated by geographically indexed observations.
• Accurate performance assessment is crucial for deploying machine learning models in real-world geographic applications.

Purpose of the Study:

• To present a tiered spatial validation workflow that distinguishes between interpolation within observed areas and transfer to distinct geographic regions.
• To provide a robust framework for evaluating the geographic generalization capabilities of machine learning models.

Main Methods:

• The workflow integrates multiple validation strategies: random folds, size-matched controls, coordinate clustering, graph-based community detection, administrative boundaries, and external regional testing.
• Performance metrics include pooled R-squared, root mean squared error, variability across random seeds, spatial autocorrelation, and sensitivity to scale.
• The approach is dataset-agnostic and demonstrated using Baltimore housing sales data.

Main Results:

• Random validation overestimated model performance compared to spatial methods like coordinate clustering and graph partitioning.
• Size-matched controls effectively ruled out training set size as a confounding factor.
• Administrative and external holdouts revealed a significant performance gap, indicating that coordinate clustering, while better than random, remains optimistic for true external transfer.

Conclusions:

• The proposed tiered spatial validation workflow provides a more realistic assessment of geographic machine learning model performance.
• It enables interpretable evaluation of the validation gap by incorporating controls, diagnostics, and regional holdouts.
• This method is essential for ensuring that performance claims accurately reflect a model's intended deployment setting.