Leveraging large language models for structured information extraction from pathology reports

Area of Science:

• Computational pathology
• Natural Language Processing
• Medical Informatics

Background:

• Structured information extraction from unstructured histopathology reports is crucial for clinical research data accessibility.
• Manual extraction is time-consuming and limits scalability.
• Large language models (LLMs) offer automated extraction via zero-shot prompting, eliminating the need for labeled data or training.

Purpose of the Study:

• To evaluate the accuracy of LLMs in extracting structured information from breast cancer histopathology reports.
• To compare LLM performance against manual extraction by a trained human annotator.

Main Methods:

• Developed the Medical Report Information Extractor web application utilizing LLMs.
• Created a gold-standard dataset for evaluation.
• Assessed five LLMs, including GPT-4o and Llama 3 models, on 111 breast cancer histopathology reports, extracting 51 pathology features.

Main Results:

• Llama 3.1 405B (94.7% accuracy) and GPT-4o (96.1%) demonstrated comparable accuracy to the human annotator (95.4%).
• Llama 3.1 70B (91.6%) performed below human accuracy but offers a viable self-hosting option due to lower computational needs.

Conclusions:

• An open-source tool for structured information extraction achieved expert human-level accuracy using state-of-the-art LLMs.
• The tool is customizable via natural language and promotes data standardization, accessibility, and interoperability for analytics.