Evaluating deep learning sepsis prediction models in ICUs under distribution shift: a multi-centre retrospective...

Sepsis prediction models trained on ICU data often fail to generalize under external validation because of distribution shift. Prior studies have focused on direct model deployment or conventional transfer learning methods (e.g., fine-tuning), yet systematic exploration of alternative strategies remains limited. We quantify shifts across three harmonized adult ICU cohorts (HiRID, MIMIC-IV, eICU; 216,536 stays) and compare five deployment strategies: generalization, fine-tuning/retraining, target training, supervised domain adaptation (DA), and fusion-training, across multiple deep learning architectures, and four target-data regimes (none; small ≤ 8k; medium 8-32k; large ≥ 32k stays). Fine-tuning consistently underperforms, even though it has been the go-to method in literature. Retraining and fusion perform best in small and large target data regimes, while DA yields the most stable gains with medium target data, improving AUROC and normalized AUPRC over other methods. These results argue for moving beyond routine fine-tuning for sepsis prediction and selecting strategies by target-data availability and operational context.