Medication information extraction using local large language models.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Artificial Intelligence

Background:

• Extracting medication information from unstructured clinical text is vital but challenging due to manual effort and errors.
• Current automation methods face constraints like expertise demands, time limits, IT infrastructure, and transparency needs.
• Generative large language models (LLMs) and parameter-efficient fine-tuning offer promising solutions.

Purpose of the Study:

• To evaluate local LLMs for automated, end-to-end medication information extraction.
• To assess the performance of fine-tuned LLMs on both English and German clinical datasets.
• To enhance prediction transparency using explainability techniques.

Main Methods:

• Utilized named entity recognition and relation extraction with local LLMs.
• Employed format-restricting instructions and an automated feedback pipeline for evaluation.
• Applied token-level Shapley values for visualizing and quantifying token contributions.

Main Results:

• Fine-tuned Llama models achieved new state-of-the-art results on English data, improving F1-scores by up to 10 pp. for adverse drug events and 6 pp. for medication reasons.
• Llama established a new benchmark on the German dataset, outperforming traditional methods by up to 16 pp. micro average F1-score.
• OpenBioLLM showed limitations in structured outputs and experienced hallucinations.

Conclusions:

• Fine-tuned local open-source generative LLMs surpass current state-of-the-art methods for medication information extraction.
• These models provide high performance with limited resources in clinical settings, effective in both English and German.
• Shapley values enhance prediction transparency, aiding clinical decision-making.