Censoring-Aware Reinforcement Learning to Optimize Early Risk Alerts from Longitudinal Clinical Data

Early recognition of chronic conditions is critical to ensure patients receive timely interventions and support. Passive surveillance of routine electronic health records (EHRs) provides information about longitudinal health trajectories that can support prompt recognition and inform associated early actions. However, relevant information is acquired at a different rate for each patient, and there is an inherent trade-off between the earliness versus the specificity of diagnosis and related actions. Therefore, determining when to alert providers about a likely chronic condition requires us to weigh the predicted risk at the given time against the anticipated value of future information. To address this challenge, we analyze the optimal timing of early alerts using a Partially Observable Markov Decision Process (POMDP) with asymmetric reward. To learn an optimal alerting policy, we then propose a model-free reinforcement learning (RL) framework tailored to long-term clinical event surveillance from EHRs. Our proposed framework overcomes the pervasive issue of right-censoring in offline EHRs by leveraging a pseudo-label imputation approach. We also analytically demonstrate that entropy-regularized RL enables post-hoc threshold calibration to adapt the learned policy to specific preferences regarding the importance of earliness versus specificity without retraining. Systematic evaluations in synthetic data reveal that the advantage of RL-based look-ahead planning is maximized when diagnostic evidence emerges in predictable information bursts. Finally, real-world validations on two clinical cohorts show that our policy achieves an actionable lead time of 20.9 months prior to Alzheimer's disease diagnosis and 8.7 months prior to autism diagnosis in a pediatric cohort, both at 90% specificity. Our method and results provide a generalizable blueprint for optimal surveillance of chronic disease processes.