Span-based annotation framework for LLM-based clinical named entity recognition: development and validation using...

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence

Background:

• Clinical Named Entity Recognition (NER) is crucial for extracting information from electronic health records.
• Large Language Models (LLMs) show potential for improving clinical NER tasks.
• Developing effective annotation frameworks is essential for training accurate clinical NER models.

Purpose of the Study:

• To develop and validate a span-based annotation framework for clinical NER using LLMs.
• To evaluate the performance of fine-tuned LLMs against baseline models and human annotators.
• To assess the robustness of the framework in a real-world clinical setting.

Main Methods:

• Constructed two datasets with word-level and phrase-level annotations from Korean emergency department clinical notes.
• Fine-tuned a Korean language-specific LLM on these datasets, creating three variants.
• Compared fine-tuned LLM variants against few-shot LLM and fine-tuned small language model (SLM) baselines.

Main Results:

• All three fine-tuned LLM variants significantly outperformed the baseline models in clinical NER tasks.
• The final LLM variant achieved F1 scores exceeding 0.80 across various evaluation metrics.
• The fine-tuned LLM demonstrated superior performance compared to a human annotator.

Conclusions:

• Supervised fine-tuning (SFT) is superior to prompt engineering for LLM-based clinical NER.
• The developed span-based annotation framework and fine-tuned LLM are robust and applicable in practical settings.
• LLMs can effectively handle complex and long entity spans in multilingual clinical data.