A Machine Learning Algorithm to Predict Medical Device Recall by the Food and Drug Administration

Medical device recalls are important to the practice of emergency medicine, as unsafe devices include many ubiquitous items in emergency care, such as vascular access devices, ventilators, infusion pumps, video laryngoscopes, pulse oximetry sensors, and implantable cardioverter defibrillators. Identification of dangerous medical devices as early as possible is necessary to minimize patient harms while avoiding false positives to prevent removal of safe devices from use. While the United States Food and Drug Administration (FDA) employs an adverse event reporting program (MedWatch) and database (MAUDE), other data sources and methods might have utility to identify potentially dangerous medical devices. Our objective was to evaluate the sensitivity, specificity, and accuracy of a machine learning (ML) algorithm using publicly available data to predict medical device recalls by the FDA.

We identified recalled medical devices (RMD) and non-recalled medical devices (NRMD) using the FDA's website and online database. We constructed an ML algorithm (random forest regressor) that automatically searched Google Trends and PubMed for the RMDs and NRMDs. The algorithm was trained using 400 randomly selected devices and then tested using 100 unique random devices. The algorithm output a continuous value (0-1) for recall probability for each device, which were rounded for dichotomous analysis. We determined sensitivity, specificity, and accuracy for each of three time periods prior to recall (T-3, 6, or 12 months), using FDA recall status as the reference standard. The study adhered to relevant items of the Standards for Reporting Diagnostic accuracy studies (STARD) guidelines.

Using a rounding threshold of 0.5, sensitivities for T-3, T-6, and T-12 were 89% (95% confidence interval [CI] 69-97), 90% (95% CI 70-97), and 75% (95% CI 53-89). Specificity was 100% (95% CI 95-100) for all three time periods. Accuracy was 98% (95% CI 93-99) for T-3 and T-6, and 95% (95% CI 89-99) for T-12. Using tailored thresholds yielded similar results.

An ML algorithm accurately predicted medical device recall status by the FDA with lead times as great as 12 months. Future research could incorporate longer lead times and data sources including FDA reports and prospectively test the ability of ML algorithms to predict FDA recall.