Evaluation of Medical Problem Extraction from Electronic Clinical Documents Using MetaMap Transfer (MMTx)

To improve the use and quality of the electronic Problem List, which is at the heart of the problem-oriented medical record in development in our institution (Intermountain Health Care, Utah, U.S.), we developed an Automated Problem List system using Natural Language Processing (NLP) technologies. A key part of this system is a module that automatically extracts potential medical problems from free-text clinical documents. The NLP module uses MMTx, developed at the U.S. National Library of Medicine. Negation detection was added to this application by adapting a negation detection algorithm called NegEx. To evaluate the adequacy of the performance of the NLP module for our Automated Problem List system, we evaluated it with 160 electronic clinical documents of different types. Two different data sets for MMTx were used: the default full UMLS data set and a customised subset adapted to detect the set of 80 medical problems we are interested in. With the default data set, we measured a recall of 0.74 (95% CI 0.68-0.8) and a precision of 0.76 (0.69-0.82). The customised subset had a significantly better recall of 0.9 (0.85-0.94), and a non-significantly different precision of 0.69 (0.63-0.75).