Open-Source Hybrid Large Language Model Integrated System for Extraction of Breast Cancer Treatment Pathway From...

Area of Science:

• Medical Informatics
• Oncology
• Natural Language Processing

Background:

• Automated curation of breast cancer treatment data is crucial for evidence-based patient management and treatment pathway assessment.
• Challenges include complex, inconsistent clinical data and extracting information from free-text narratives.

Purpose of the Study:

• To develop and validate a hybrid information extraction framework for automated, minimal-human-involvement curation of breast cancer treatment data.
• To accurately extract longitudinal treatment timelines from time-stamped clinical notes.

Main Methods:

• A hybrid, two-phase information extraction framework combining a Unified Medical Language System parser and a fine-tuned large language model (LLM).
• The framework was trained end-to-end as a question-answering model to identify specific cancer treatments.
• Internal validation on 26,692 patients and external validation on 162 patients from different institutions.

Main Results:

• The proposed model achieved an average AUROC of 0.942 (internal) and 0.924 (external).
• Demonstrated a superior trade-off between sensitivity (79.2%) and specificity (76.2%) compared to rule-based and structured code methods.
• Out-of-the-box LLMs showed high specificity but low sensitivity for targeted clinical tasks.

Conclusions:

• The framework effectively extracts temporal cancer treatment information from diverse clinical notes, irrespective of treatment setting or time frame.
• The code is packaged as a Docker image for easy deployment and shared under an open-source academic license to support the research community.