Signal detection of adverse events in medical devices using natural language processing: a case study in pelvic mesh

Disproportionality analysis is used to detect safety signals for post-market surveillance from adverse events reported to regulatory bodies but is challenging when reports contain unstructured free-text. We implemented a proof-of-concept system combining natural language processing of free-text data with disproportionality analysis, using a known safety signal from pelvic mesh. Free-text reports in an Australian spontaneous adverse event report database between 2012 and 2017 were classified using topic modelling. 'Pain' was the most frequent clinical topic and was a known adverse event from pelvic mesh. Signal detection with three different comparators (hernia mesh, hernia and other mesh and all other devices) was performed every quarter with three disproportionality methods (Proportional Reporting Ratio, Bayesian Confidence Propagation Neural Network and maximised Sequential Probability Ratio Test). All methods adjusted the Type I error threshold for multiple looks at the data. A safety signal for pelvic mesh compared with hernia and other mesh was detected by all three methods in 3rd quarter of 2014, three years before the device was withdrawn from Australia in November 2017. Bayesian Confidence Propagation Neural Network most reliably accounted for uncertainty in the pain classification. Further investigation is required with other devices and databases to validate our proof-of-concept system.