Leveraging Large Language Models for Thyroid Nodule Information Extraction and Matching Across Medical Reports.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Oncology Data Analysis

Background:

• Manual extraction of thyroid nodule features from clinical reports is time-consuming and inconsistent.
• Accurate feature extraction is vital for patient management decisions, including surgery or surveillance.
• Electronic health record data presents challenges for efficient information retrieval.

Purpose of the Study:

• To evaluate the efficacy of open-source large language models (LLMs) in automating thyroid nodule feature extraction and matching.
• To compare LLM performance against manual annotations using a retrospective dataset.
• To assess the potential of LLMs in reducing clinical workload and inter-rater variability.

Main Methods:

• Developed an annotation schema for thyroid nodule characteristics from ultrasound and pathology reports.
• Utilized a retrospective dataset comprising 451 ultrasound and pathology report pairs.
• Benchmarked two LLMs, Llama-3.3 70B and QwQ-32B, against manual extraction and annotation.

Main Results:

• Both evaluated LLMs demonstrated high accuracy in extracting key thyroid nodule features (e.g., location, size, biopsy results).
• QwQ-32B achieved a superior F1 score of 0.987 for the intricate task of matching nodules across different reports.
• LLM performance closely approximated manual annotation accuracy for critical clinical data.

Conclusions:

• Open-source LLMs show significant promise for automating the extraction and matching of thyroid nodule features from clinical reports.
• Integration of LLMs into clinical workflows can substantially decrease clinician workload and improve consistency.
• LLM-driven automation maintains high accuracy, supporting informed patient management decisions for thyroid nodules.