Local Large Language Models for Complex Structured Tasks.

Area of Science:

• Natural Language Processing
• Medical Informatics
• Machine Learning

Background:

• Extracting structured data from unstructured clinical text is challenging.
• Large language models (LLMs) offer advanced language understanding but often require extensive resources.
• Local training of LLMs can optimize performance for domain-specific tasks.

Purpose of the Study:

• To develop and evaluate an approach combining LLM reasoning with local training for structured data extraction.
• To extract condition codes from surgical pathology reports using fine-tuned LLMs.
• To compare the performance of different LLM architectures on this task.

Main Methods:

• Utilized over 150,000 uncurated surgical pathology reports.
• Fine-tuned local LLMs, including LLaMA, BERT, and LongFormer, for generative instruction following.
• Evaluated models on extracting structured condition codes from gross descriptions, diagnoses, and associated codes.

Main Results:

• LLaMA-based models significantly outperformed BERT-style models across all metrics.
• LLaMA models demonstrated superior performance with large datasets for complex, multi-label condition code extraction.
• The approach successfully generated structured outputs from domain-specific medical language.

Conclusions:

• Combining LLMs with local training provides an effective method for structured generative tasks in the medical domain.
• LLaMA models are highly capable for complex, large-scale medical text analysis and condition code extraction.
• This approach facilitates the utilization of LLMs for specialized, data-intensive clinical applications.