Assessing the utility of natural language processing for detecting postoperative complications from free medical text

Postoperative complication rates are often assessed through administrative data, although this method has proven to be imprecise. Recently, new developments in natural language processing have shown promise in detecting specific phenotypes from free medical text. Using the clinical challenge of extracting four specific and frequently undercoded postoperative complications (pneumonia, urinary tract infection, sepsis, and septic shock), it was hypothesized that natural language processing would capture postoperative complications on a par with human-level curation from electronic health record free medical text.

Electronic health record data were extracted for surgical cases (across 11 surgical sub-specialties) from 18 hospitals in the Capital and Zealand regions of Denmark that were performed between May 2016 and November 2021. The data set was split into training/validation/test sets (30.0%/48.0%/22.0%). Model performance was compared with administrative data and manual extraction of the test data set.

Data were obtained for 17 486 surgical cases. Natural language processing achieved a receiver operating characteristic area under the curve of 0.989 for urinary tract infection, 0.993 for pneumonia, 0.992 for sepsis, and 0.998 for septic shock, whereas administrative data achieved a receiver operating characteristic area under the curve of 0.595 for urinary tract infection, 0.624 for pneumonia, 0.571 for sepsis, and 0.625 for septic shock.

The natural language processing approach was able to capture complications with acceptable performance, which was superior to administrative data. In addition, the model performance approached that of manual curation and thereby offers a potential pathway for complete real-time coverage of postoperative complications across surgical procedures based on natural language processing assessment of electronic health record free medical text.