Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study

Area of Science:

• Clinical Natural Language Processing (NLP)
• Artificial Intelligence in Healthcare

Background:

• Large language models (LLMs) offer potential for clinical NLP tasks like medical question answering (MQA).
• Concerns regarding cost, computing power, and patient privacy limit LLM adoption in healthcare.
• Open-source LLMs deployed within institutional firewalls can mitigate privacy risks.

Purpose of the Study:

• To evaluate the performance of a locally deployed LLM for automated MQA from surgical pathology reports.
• To compare the accuracy and efficiency of an LLM against human reviewers in extracting clinical information.

Main Methods:

• 84 thyroid cancer surgical pathology reports were analyzed by two human reviewers and the FastChat-T5 LLM.
• Reports were segmented, converted to embeddings, and contextually integrated for LLM processing.
• Twelve medical questions were posed to extract staging and recurrence risk data, with response time and concordance evaluated.

Main Results:

• Human reviewers achieved 99% concordance.
• The LLM achieved 89% concordance with human reviewers.
• The LLM responded significantly faster (19.56 minutes) compared to human reviewers (170.7 and 115 minutes).

Conclusions:

• Locally deployed LLMs offer substantial time savings for MQA in clinical settings with acceptable accuracy.
• Prompt engineering and fine-tuning can further improve automated data extraction from clinical narratives.
• LLMs hold promise for real-time clinical insights when privacy is ensured.