Zero-Shot LLMs for Named Entity Recognition: Targeting Cardiac Function Indicators in German Clinical Texts

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• Large Language Models (LLMs) show promise for medical applications.
• Open-source LLMs (Llama 3, Gemma, Mistral, Mixtral) are explored for non-English clinical text analysis.
• A research gap exists for extracting medical parameters from German clinical texts.

Purpose of the Study:

• Evaluate open-source LLMs for medical parameter extraction from German clinical texts.
• Focus on cardiovascular function indicators from cardiac MRI reports.
• Assess performance of Llama 3, Gemma, Mistral, and Mixtral.

Main Methods:

• Extracted 14 cardiovascular indicators (e.g., LV-EF, RV-EF) from 497 cardiac MRI reports.
• Assessed model performance using right annotation and Named Entity Recognition (NER) accuracy.
• Utilized open-source LLMs without specific fine-tuning for German or data extraction.

Main Results:

• Achieved high performance, with up to 95.4% right annotation accuracy.
• Reached up to 99.8% Named Entity Recognition (NER) accuracy.
• Demonstrated effectiveness despite lack of specialized fine-tuning.

Conclusions:

• Open-source LLMs are effective for extracting medical parameters from clinical texts.
• High accuracy is achievable even for non-English texts like German.
• Recommends using LLMs for medical data extraction due to accuracy and efficiency.