Beyond the Model: Practical Insights from Monitoring Predictive Models across Diverse Clinical Workflows

To support the artificial intelligence (AI) lifecycle in an integrated academic health system, we implemented a modular monitoring system to oversee electronic health record (EHR) vendor-provided clinical predictive models. This case study describes the lessons learned using this modular system to support sustainable oversight of the deployed predictive models across their lifecycle.

We developed a modular monitoring system using automated data pipelines refreshed daily to support longitudinal oversight of four clinical predictive models developed by our EHR vendor (Epic Systems). An interactive monitoring application was designed to bridge the technical-operational gap for stakeholders involved in AI lifecycle decisions. Interactive reports include automated and assisted threshold-independent and threshold-dependent performance measures, flag rates, and problem-specific metrics crafted to support operational decision-making.

Through the lenses of people, process, and technology, we describe lessons learned from the real-world implementation of a monitoring system. Effective oversight required clearly defined dyadic ownership (technical and operational) and embedded monitoring activities within existing AI governance decision-making processes across all AI lifecycle stages. Monitoring extended beyond performance drift to include alert burden, workflow alignment, and problem-level signals. Modularity enabled rapid adaptation as models and workflows evolved. Silent alert simulations proved valuable for selecting preimplementation thresholds that aligned with clinical capacity and workflows but were limited in predicting postimplementation performance when the models faced real-world human-alert interactions.

Implementing a modular, governance-aligned monitoring system enabled sustained oversight of vendor-provided clinical predictive models and shifted monitoring beyond traditional technical performance metrics toward user-centric and operationally meaningful measures. These findings highlight the importance of integrated monitoring infrastructure as a core component of responsible AI governance in clinical settings.