Exploring Prompt-Based Large Language Model (LLM) Approach for Medication Error-Related Named Entity Recognition in...

Area of Science:

• Natural Language Processing (NLP)
• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Medication errors pose significant risks in healthcare.
• Innovative analytical methods are crucial for identifying and mitigating these errors.
• Named Entity Recognition (NER) is vital for extracting critical information from unstructured medical text.

Purpose of the Study:

• To explore the efficacy of generative pre-trained language models (LLMs) for NER in Japanese medical incident reports.
• To compare the performance of various LLMs against established models and benchmark datasets.
• To propose a prompt-based framework for addressing clinical NER challenges.

Main Methods:

• Evaluation of four LLMs (Llama-3-ELYZA, BioMistral-7B, GPT-4.0 mini, GPT-4.0) on a national open-source dataset of Japanese medical incident reports.
• Comparison of LLM NER performance against a previously published annotated dataset.
• Implementation of a prompt-based framework, including entity type definitions and few-shot examples.

Main Results:

• GPT-4.0 demonstrated superior performance among the tested LLMs but did not exceed the performance of a fine-tuned BERT model.
• Few-shot prompting achieved high accuracy for numerical entities (e.g., 'Strength_rate', F1-score: 0.951).
• Performance on clinically specific entities was suboptimal due to linguistic complexities, though providing definitions and examples improved GPT-4.0's results.

Conclusions:

• Generative LLMs offer potential for clinical NER without extensive fine-tuning.
• Linguistic complexities in clinical data present challenges for current LLM-based NER.
• Further research is needed to optimize LLM performance for specialized medical terminology and complex clinical contexts.