Deep continual multitask out-of-hospital incident severity assessment from changing clinical features

When developing machine learning models to support emergency medical triage, it is important to consider how changes over time in the input features can negatively affect the models' performance. The objective of this study was to assess the effectiveness of novel deep continual learning pipelines in maximizing model performance when input features change over time, including the emergence of new features and the disappearance of existing ones.

The model is designed to identify life-threatening situations, predict their admissible response delay, and determine their institutional jurisdiction. We analyzed a total of 1 414 575 events spanning from 2009 to 2019. We provided empirical evidence for covariate shifts and measured their negative effects on model performance. Then, we proposed a set of continual learning strategies to deal with 1) changing feature domains and 2) parameter updating over time. For changing feature domains, we proposed and assessed a static domain approach, a dynamic domain approach, and a predefined approach. For parameter updating over time, we designed and evaluated five approaches: from-scratch, fine-tuning, a cumulative, rehearsal and Elastic Weight Consolidation (EWC).

Our findings demonstrate performance improvements, with the best strategy combination-dynamic feature domain coupled with EWC-yielding gains of up to 5.9% in life-threatening situations, 18.6% in response delay, and 2.4% in jurisdiction, in absolute F1-score compared to the current triage protocol. Additionally, we observed improvements of up to 5.4% in life-threatening situations and 11% in response delay, in absolute F1-score, compared to non-continual approaches.

The methods proposed in this work improve performance by mitigating the negative effects of covariate shifts.