Leveraging Large Language Models for Real-World Data Evidence: A Framework for Automated Treatment Extraction and...

Area of Science:

• Oncology
• Natural Language Processing
• Health Informatics

Background:

• Collecting comprehensive cancer treatment data from medical records is crucial for real-world evidence studies.
• Unstructured clinical text hinders systematic, high-quality treatment information extraction.
• Manual data extraction is time-consuming, necessitating automated solutions.

Purpose of the Study:

• To evaluate the utility of Llama family large language models (LLMs) for automated oncology treatment information extraction.
• To guide researchers in utilizing cancer registry data for insights beyond clinical trials.

Main Methods:

• Four instruction-tuned Llama models (1B, 3B, 8B, 70B parameters) were assessed for treatment extraction from clinical documents.
• A unified oncology knowledge base was developed for standardizing extracted entities.
• Performance was measured using accuracy metrics (Precision, Recall, F1-Score) and operational feasibility.

Main Results:

• A positive correlation between model size and extraction accuracy was observed, with F1-scores ranging from 0.609 (1B) to 0.828 (70B).
• Larger models showed higher accuracy and compliance but incurred greater computational costs.
• The 8B model demonstrated strong performance, with diminishing returns noted for the 70B model.

Conclusions:

• LLMs are a viable technology for automating oncology treatment extraction.
• The 8B-parameter LLM provides an effective balance of accuracy and computational efficiency.
• Standardized data integration via a knowledge base enhances real-world evidence analyses.