Leveraging open-source large language models for clinical information extraction in resource-constrained settings.

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence (AI)
• Clinical Informatics

Background:

• Clinical information extraction from Dutch medical reports is challenging due to language barriers and limited resources.
• Open-source generative large language models (LLMs) offer potential solutions for automated text analysis.

Purpose of the Study:

• To evaluate the zero-shot performance of open-source generative LLMs on clinical information extraction from Dutch medical reports.
• To assess LLM capabilities across classification, regression, and named entity recognition (NER) tasks using the DRAGON benchmark.

Main Methods:

• Developed the llm_extractinator framework for scalable, open-source clinical text information extraction.
• Evaluated 9 multilingual LLMs on 28 DRAGON benchmark tasks in a zero-shot setting.
• Investigated the impact of in-context translation to English on model performance.

Main Results:

• Llama-3.3-70B achieved the highest utility score (0.760), with other models like Phi-4-14B and Qwen-2.5-14B also performing well.
• LLMs outperformed or matched a fine-tuned RoBERTa baseline on 17 tasks, particularly in regression and structured classification.
• Named entity recognition (NER) performance was consistently low across all models, and translation to English generally reduced performance.

Conclusions:

• Open-source generative LLMs demonstrate significant zero-shot capabilities for clinical NLP tasks, especially in structured inference.
• Models around 14B parameters offer a good balance of performance and computational cost, though larger models lead.
• Native language support is crucial, as translation to English negatively impacts performance, highlighting the value for low-resource and multilingual settings.