Evaluating a candidate predictor beyond statistical significance: Atmospheric pressure and psychiatric day-care attendance as a worked example

In observational research, statistical detectability, robustness to adjustment, and predictive utility are distinct properties of a candidate predictor, yet they are often conflated. A statistically significant association may weaken after accounting for co-varying factors and may add little to out-of-sample prediction. We illustrate this distinction using atmospheric pressure and daily attendance at a Japanese psychiatric day-care as a worked example. Using four consecutive fiscal years of facility-level daily attendance, we evaluated same-day atmospheric pressure as a pre-specified candidate predictor in three stages: statistical detectability, robustness to adjustment for co-varying meteorological factors and seasonality, and incremental out-of-sample predictive utility beyond institutional scheduling variables and recent attendance history. Same-day pressure was positively associated with attendance in all four years (incidence rate ratios 1.003-1.007 per hPa). However, effect-estimate stability varied across years, with marked attenuation in FY2022, moderate attenuation after precipitation adjustment in FY2023, and specification-sensitive precision in FY2024, whereas FY2025 was comparatively stable. In contrast, the structural baseline incorporating scheduling variables and recent attendance history provided strong out-of-sample prediction, explaining 55%-69% of the variance across years. Adding atmospheric pressure produced only small and unstable changes in the primary expanding-window cross-validated performance (mean ΔR2 ≈ +0.005), with increments varying in sign across years. Thus, a candidate predictor that was repeatedly statistically detectable showed limited stability of effect magnitude or precision under reasonable adjustment and little incremental predictive utility beyond a strong structural baseline. In this highly schedule-driven, single-facility setting, the weather signal examined contributed little incremental information for out-of-sample prediction of aggregate attendance. More broadly, this worked example illustrates why candidate predictors should be evaluated separately for detectability, robustness of effect estimates, and out-of-sample predictive utility before statistically significant associations are interpreted as meaningful or actionable.