Internal Validation of a Machine Learning-Based CDSS for Antimicrobial Stewardship

Background: Antimicrobial stewardship programs (ASPs) are essential in combating antimicrobial resistance (AMR); however, limited resources hinder their implementation. Arkstone, a biotechnology company, developed a machine learning (ML)-driven clinical decision support system (CDSS) to guide antimicrobial prescribing. While AI (artificial intelligence) applications are increasingly used, each model must be carefully validated. Methods: Three components of the ML system were assessed: (1) A prospective observational study tested its ability to distinguish trained from novel data using various validation techniques and BioFire molecular panel inputs. (2) An anonymous retrospective analysis of internal infectious disease lab results evaluated the recognition of novel versus trained complex datasets. (3) A prospective observational validation study reviewed clinical recommendations against standard guidelines by independent clinicians. Results: The system achieved 100% accuracy (F1 = 1) in identifying 111 unique novel data points across 1110 tests over nine training sessions. It correctly identified all 519 fully trained and 644 novel complex datasets. Among 644 clinician-trained reports, there were no major discrepancies in recommendations, with only 100 showing minor differences. Conclusions: This novel ML system demonstrated high accuracy in distinguishing trained from novel data and produced recommendations consistent with clinical guidelines. These results support its value in strengthening CDSS and ASP efforts.