An Entity Extraction Pipeline for Medical Text Records Using Large Language Models: Analytical Study.

Area of Science:

• Clinical Informatics
• Natural Language Processing
• Artificial Intelligence in Healthcare

Background:

• Clinical data, including text, is crucial for understanding disease progression.
• Large language models (LLMs) offer potential for semantic feature extraction from clinical text.
• Security and feature hallucination are key challenges with LLMs in clinical research.

Purpose of the Study:

• To introduce a novel modular pipeline for semantic feature extraction from textual patient admission records using LLMs.
• To evaluate the pipeline's performance with different LLMs and on consumer-grade hardware.

Main Methods:

• A systematic pipeline involving concept extraction, aggregation, question generation, corpus extraction, and question-and-answer scale extraction was designed.
• The pipeline was tested using Qwen-14B-Chat (QWEN) and Baichuan2-13B-Chat (BAICHUAN) LLMs on 25,709 pregnancy cases.
• Evaluation metrics included accuracy, precision, null ratio, and time consumption, with performance also assessed on a quantified LLM version on a consumer-grade GPU.

Main Results:

• The pipeline achieved high precision, with Qwen-14B-Chat showing 95.52% accuracy and 92.93% precision, and Baichuan2-13B-Chat showing 95.86% accuracy and 90.08% precision.
• Low null ratios were observed, and time consumption varied.
• An INT4-quantified version of Qwen-14B-Chat achieved 97.28% accuracy and a 0% null ratio, demonstrating enhanced performance.

Conclusions:

• The pipeline demonstrated consistent performance across different LLMs for efficient clinical feature extraction from textual data.
• Reliable performance was observed on consumer-grade hardware.
• This approach provides an effective solution for mining clinical research data from unstructured text records.