Employing Consensus-Based Reasoning with Locally Deployed LLMs for Enabling Structured Data Extraction from Surgical...

Area of Science:

• Computational pathology
• Artificial intelligence in healthcare
• Natural language processing for clinical data

Background:

• Surgical pathology reports are crucial for cancer diagnosis, staging, and treatment planning.
• The free-text nature and variability of these reports hinder automated data extraction.
• Accurate extraction of diagnostic variables is essential for cancer registries and clinical decision-making.

Purpose of the Study:

• To develop and evaluate a consensus-driven, reasoning-based framework using locally deployed large language models (LLMs) for extracting key diagnostic variables from surgical pathology reports.
• To assess the accuracy and interpretability of LLM-driven data extraction across diverse tumor types and institutions.
• To provide a transparent and auditable solution for integrating AI into pathology workflows.

Main Methods:

• A framework employing multiple locally deployed LLMs to extract six key variables: site, laterality, histology, stage, grade, and behavior.
• Each LLM output included justifications, which were evaluated by a separate reasoning model for accuracy and coherence.
• Consensus values were aggregated, and expert pathologists validated the results on over 4,000 reports from TCGA and Moffitt Cancer Center.

Main Results:

• High agreement was achieved in the TCGA dataset for behavior (100.0%), histology (98.5%), site (95.2%), and grade (95.6%).
• Performance for stage (87.6%) and laterality (84.8%) was lower in TCGA.
• Pathology reports from Moffitt (brain, breast, lung) showed high accuracy for histology (95.6%), behavior (98.3%), and stage (92.4%).
• Challenges included inconsistent sentinel lymph node details and anatomical ambiguity.
• Statistical analysis revealed significant effects of model type, variable, and organ system on performance.

Conclusions:

• Locally deployed LLMs, within a consensus-based framework, offer a transparent, accurate, and auditable solution for extracting critical diagnostic information from pathology reports.
• This AI-driven approach can improve cancer registry abstraction and synoptic reporting.
• Stratified, multi-organ evaluation frameworks are crucial for benchmarking LLMs in clinical applications.