Continuous Value Tokenization Improves Medical Event Foundation Models

Medical foundation models convert patient records into token sequences for autoregressive prediction, but numeric values such as lab results, vital signs, and time intervals are typically discretized into bins, losing precision and misaligning with clinical thresholds. We trained decoder-only transformer models (47 million parameters) on MIMIC-IV data (364,627 patients; 375 million observations) to compare three tokenization strategies: Discrete (binned values), Continuous Factored (continuous values preserving sequence length), and Continuous Fused (continuous values fused with measurement-type tokens). We evaluated next-token prediction, numeric value prediction, and three clinical tasks: ED disposition at triage, ICD code prediction, and DRG prediction at discharge. Continuous Fused tokenization reduced median sequence length by 34%, reached the Discrete model's final next-token loss in 30% of training iterations, and improved numeric prediction accuracy by 30.25% median nRMSE reduction. ICD code prediction favored Continuous Fused (AU-PRC 0.457 vs. 0.446; p < 0.001); DRG prediction was equivalent between Continuous Fused and Discrete; ED disposition accuracy was equivalent across all models (∼0.900), though Discrete achieved better calibration. We additionally explain why predictive performance improves with Monte Carlo sample count and derive a scaling law to predict performance gains from increasing simulation budget. Continuous-value tokenization offers substantial efficiency and precision gains while maintaining comparable clinical task performance, with no modifications to the standard transformer architecture.

Medical event foundation models are increasingly used to forecast clinical outcomes. These models predict using patient records as sequences of "tokens," much like a language model predicts next words. Numerical values such as lab results and vital signs pose a problem: they are usually grouped into discrete bins before being tokenized, which loses precision and ignores the fact that medical decisions often depend on specific numerical thresholds. We compared three ways of representing continuous values in medical foundation models: the standard binning approach and two new approaches that preserve the actual values. The continuous-value approaches trained about three times faster, produced 34% shorter sequences, and achieved comparable accuracy on clinical prediction tasks including emergency department triage, ICD diagnosis codes, and hospital billing codes. These gains require no modifications to the standard transformer architecture. We also discovered that commonly used evaluation metrics exhibit a systematic bias depending on the number of simulated predictions, and that this bias follows a predictable mathematical pattern, enabling researchers to estimate full-scale performance from smaller, less expensive simulation experiments.