Limits of Generative Pre-Training in Structured EMR Trajectories with Irregular Sampling

Foundation models are commonly trained using autoregressive pretraining and are increasingly applied to structured electronic medical records (EMRs) for clinical prediction. However, autoregressive pre-training was originally developed for sequence generation in text, and its suitability for capturing longitudinal clinical context in structured, irregularly sampled EMRs remains insufficiently evaluated. In this work, we return to first principles by examining the patient trajectories generated by autoregressive models. We train two models - a sequence-to-sequence LSTM and a reduced ETHOS-style Transformer - that share core architectural and training principles with state-of-the-art EHR foundation models. Using longitudinal antiretroviral therapy (ART) for HIV and acute hypotension co-horts, we evaluate the goodness-of-fit of generated trajectories via distributional and correlational fidelity, focusing on whether clinically meaningful relationships between variables are preserved over time. We find that cross-feature coherence degrades under irregular sampling, indicating that autoregressive models may not reliably capture dynamic patient context. These findings highlight the need for domain-specific, generation-based evaluation before relying on autoregressive pre-training as a foundation for transfer learning in healthcare.