Development and Assessment of a Pipeline for Extracting Structured Data From Free-Text Medical Reports Using a Large...

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Oncology

Background:

• Pathology reports contain crucial clinical data for breast cancer (BC) but are difficult to structure at scale.
• Traditional natural language processing (NLP) methods require extensive manual annotation and training.
• Large Language Models (LLMs) offer a potential solution for automated data extraction.

Purpose of the Study:

• To investigate the efficacy of Mistral Large LLM for automatically extracting three key breast cancer (BC) biomarkers from unstructured pathology reports.
• To evaluate the performance of an LLM-based pipeline for structuring clinical data.

Main Methods:

• Developed and assessed a pipeline combining the Mistral Large LLM with a postprocessing phase.
• Evaluated performance on two datasets: 1,152 BC pathology reports and a gold standard database of 101 metastatic BC patients.
• Explored the impact of confidence prompts (CP), chain-of-thought (CoT), and few-shot examples on pipeline performance.

Main Results:

• Achieved document-level F1 scores >95% and recall/precision >94% for estrogen receptor, progesterone receptor, and HER2 status/score.
• Patient-level F1 scores ranged from 87%-90%, with recall between 83%-87% and precision >90%.
• Pipeline performance was consistent regardless of CP, CoT, or few-shot example inclusion.

Conclusions:

• Demonstrated the significant potential of LLMs like Mistral Large for extracting structured breast cancer biomarker data from pathology reports.
• Highlighted the utility of LLM-based approaches for the digital transformation of healthcare documents.