Evaluation of Large-Language Models for Structured Feature Extraction of Anatomic and Clinical Pathology Reports

Area of Science:

• Medical Informatics
• Natural Language Processing
• Computational Pathology

Background:

• Manual feature extraction from clinical notes is time-consuming and expensive.
• Natural language processing (NLP) advancements offer automated solutions for high-throughput feature extraction.
• Large language models (LLMs) show promise for structured feature extraction in pathology.

Purpose of the Study:

• To assess the accuracy of LLMs for structured feature extraction from clinical and anatomic pathology notes.
• To compare LLM performance against expert clinician labels across diverse pathology datasets.
• To develop a tool for rapid prototyping of structured function calls to LLMs.

Main Methods:

• Evaluated OpenAI GPT-4o and GPT-5 models on cardiac transplant pathology, hemoglobin variant, and urine drug test reports.
• Compared LLM-extracted features with manual labels from expert clinicians.
• Developed a web application ('toolbuilder') for designing structured function calls to LLMs.

Main Results:

• LLMs achieved high accuracy, with error rates around 5% for simple and 10% for complex cases.
• Expert-LLM concordance was high (κ>0.9), closely approaching inter-expert agreement.
• Model performance varied by type but was less sensitive to prompt engineering; errors often involved distinguishing negative from indeterminate findings.

Conclusions:

• LLM-based tools can automate time- and cost-intensive feature extraction from clinical notes.
• These automated methods hold significant value for clinical care and research.
• LLMs show potential to streamline pathology data annotation and analysis.