Iterative refinement and goal articulation to optimize large language models for clinical information extraction

Area of Science:

• Computational pathology
• Medical informatics
• Artificial intelligence in healthcare

Background:

• Extracting structured data from unstructured medical records is challenging.
• Traditional methods are insufficient for complex clinical data like pathology reports.

Purpose of the Study:

• To develop an end-to-end pipeline using large language models (LLMs) for accurate information extraction and normalization from pathology reports.
• To focus initially on kidney tumor data extraction and demonstrate adaptability to other cancer types.

Main Methods:

• Developed a novel pipeline leveraging LLMs with flexible prompt templates.
• Generated analysis-ready tabular data directly from reports.
• Implemented a human-in-the-loop iterative refinement process with an error ontology.
• Validated the pipeline on 2297 kidney tumor reports and publicly available breast and prostate cancer reports.

Main Results:

• Achieved a macro-averaged F1 score of 0.99 for kidney tumor subtypes.
• Achieved a macro-averaged F1 score of 0.97 for detecting kidney metastasis.
• Demonstrated flexibility with multiple LLM backbones and adaptability to new cancer domains.

Conclusions:

• LLM-based pipelines offer highly accurate information extraction from unstructured pathology reports.
• Successful application to kidney, breast, and prostate cancer data highlights pipeline versatility.
• Emphasizes the importance of task definition, interdisciplinary collaboration, and complexity management in clinical LLM workflows.