Can open source large language models be used for tumor documentation in Germany?-An evaluation on urological...

Area of Science:

• Medical Informatics
• Natural Language Processing

Background:

• Tumor documentation in Germany is a manual process requiring structured data extraction.
• Large language models (LLMs) offer potential for improving efficiency and reliability.
• Open source LLMs are suitable for clinical documentation due to data protection regulations.

Purpose of the Study:

• Evaluate the performance of open source LLMs in the German medical domain for tumor documentation.
• Assess LLM capabilities in interpreting specialized medical language.
• Determine the suitability of LLMs for enhancing clinical documentation processes.

Main Methods:

• Evaluated eleven open source LLMs (1-70 billion parameters).
• Used three core tasks: tumor diagnosis identification, ICD-10 code assignment, and first diagnosis date extraction.
• Utilized an annotated dataset of anonymized urology notes and explored few-shot prompting strategies.

Main Results:

• Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B demonstrated comparable performance.
• Smaller models (<7 billion parameters) and very large models showed lower or no performance gains.
• Cross-domain examples improved few-shot prompting outcomes, indicating LLM adaptability.

Conclusions:

• Open source LLMs show significant potential for automating tumor documentation.
• 7-12 billion parameter models present an optimal performance-resource efficiency balance.
• Fine-tuning and effective prompting can establish LLMs as valuable clinical documentation tools.