Prompts to Table: Specification and Iterative Refinement for Clinical Information Extraction with Large Language...

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Natural language processing

Background:

• Extracting structured data from unstructured clinical text is challenging.
• Traditional methods face limitations in complex medical domains.
• Pathology reports contain vital information for cancer diagnosis and research.

Purpose of the Study:

• To develop and validate a novel pipeline for accurate information extraction and normalization from unstructured pathology reports using large language models (LLMs).
• To focus initially on kidney tumor reports and demonstrate adaptability to other cancer types.
• To generate analysis-ready tabular data from free-text medical records at scale.

Main Methods:

• An end-to-end pipeline leveraging LLMs for information extraction and normalization.
• Flexible prompt templates and direct production of tabular data.
• A human-in-the-loop iterative refinement process guided by an error ontology.
• Validation on 2,297 kidney tumor reports and publicly available breast and prostate cancer reports.

Main Results:

• Achieved a macro-averaged F1 score of 0.99 for kidney tumor subtypes and 0.97 for kidney metastasis detection.
• Demonstrated flexibility with multiple LLM backbones.
• Showcased adaptability to new domains (breast and prostate cancer reports).

Conclusions:

• LLM-based pipelines offer a highly accurate and efficient solution for extracting structured data from complex clinical text.
• The developed pipeline successfully structures critical information from pathology reports, facilitating analysis and research.
• Emphasizes the importance of task definition, interdisciplinary collaboration, and complexity management in clinical LLM workflows.