Functional evaluation of out-of-the-box text-mining tools for data-mining tasks

Area of Science:

• Clinical Informatics
• Natural Language Processing (NLP)
• Biomedical Data Science

Background:

• The choice between fast, simple dictionary-based term recognition and advanced NLP with richer linguistic information is debated in clinical informatics.
• Systems making different speed-linguistic trade-offs are crucial for processing clinical text data.

Purpose of the Study:

• To quantify the trade-off between speed and linguistic understanding in text processing systems for clinical research.
• To compare the performance of dictionary-based and advanced NLP systems across various clinical data tasks.

Main Methods:

• Benchmarked accuracy of NCBO Annotator (dictionary-based) and REVEAL (advanced NLP) on a public dataset (2008 i2b2 Obesity Challenge).
• Applied both systems to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE).
• Evaluated system performance on three clinical research tasks: drug safety profiling, drug-drug interaction learning, and drug-indication relationship learning.

Main Results:

• No significant difference in results between NCBO Annotator and REVEAL on large datasets for the three research tasks.
• REVEAL demonstrated higher sensitivity in one subtask with smaller datasets.

Conclusions:

• Simple term recognition methods show little to no impact on accuracy compared to advanced NLP for clinical research tasks using large datasets.
• Dictionary-based methods possess superior scalability for very large datasets.
• Promoting simple, dictionary-based methods can enhance the practical adoption of NLP for population-level analyses in clinical practice.