Medical Information Extraction With NLP-Powered QABots: A Real-World Scenario

Area of Science:

• Medical Informatics
• Natural Language Processing (NLP)
• Clinical Data Management

Background:

• Computerized medical records simplify data retrieval but extracting information from unstructured documents remains challenging.
• Vast potential within clinical records is untapped due to manual data extraction limitations.
• Natural Language Processing (NLP) offers automated text-mining solutions for clinical data.

Purpose of the Study:

• To present the architecture of the NLP Extraction and Management Tool (NEMT) for efficient clinical data extraction.
• To enable automated information retrieval from unstructured clinical documents for the Virtual Dementia Institute (IVD).
• To populate a centralized REDCap database with standardized patient data from multiple hospitals.

Main Methods:

• Developed a (semi-)automated end-to-end pipeline, NEMT, integrating a Question Answering Bot (QABot).
• Defined a common Case Report Form (CRF) and minimum dataset across sixteen Italian hospitals.
• Fine-tuned the QABot on thousands of clinical examples from IVD centers; calculated Inter-Annotator Agreement.

Main Results:

• The QABot achieved high performance metrics: Exact Match (EM) of 78.1%, F1-score of 84.7%, Lenient Accuracy (LAcc) of 0.834, and Mean Reciprocal Rank (MRR) of 0.810.
• NEMT's performance surpassed ChatGPTv3.5 on EM (68.9%) and F1-score (52.5%).
• Successfully populated a database with data from thousands of Italian patients using a standardized screening procedure.

Conclusions:

• NEMT provides an efficient solution for extracting and managing information from unstructured clinical documents.
• The developed system facilitates the creation of large, standardized datasets for future medical research.
• This approach paves the way for enhanced medical information exploitation and new research studies.