Accelerating real-world data collection using large language models in rare neoplasms: a bone sarcoma example.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Oncology Data Science

Background:

• Real-world data collection in oncology is challenging due to unstructured medical notes.
• Large language models (LLMs) show promise for extracting information from free-text data.
• This study assesses small LLMs for information extraction from Polish medical notes.

Purpose of the Study:

• To evaluate the performance of multiple small LLMs as information extractors on Polish medical notes.
• To determine the effectiveness of different prompting techniques and ensemble methods for data extraction.
• To assess the feasibility of automating data extraction from electronic health records (EHRs) in a non-English setting.

Main Methods:

• Utilized EHRs from 302 bone sarcoma patients (2016-2022).
• Annotated five key variables: pathology type, tumor size, localization, grade, and primary resection.
• Employed four small LLMs with multiple prompting techniques and an ensemble voting strategy.

Main Results:

• Single-model accuracy ranged from 17.5% to 30.3%, highly dependent on prompts.
• Tumor localization was the easiest variable to extract (up to 36.2% accuracy).
• The ensemble voting approach significantly boosted overall accuracy to 83.6%, reaching 90.0% for resection type.

Conclusions:

• Lightweight LLMs show potential for automating data extraction from medical notes, accelerating clinical research.
• Individual small LLMs are insufficient for real-world, non-English applications.
• Prompt engineering and ensemble methods are crucial for improving LLM performance in medical data extraction.