Efficient medical NER with limited data: Enhancing LLM performance through annotation guidelines.

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence (AI)
• Computational Linguistics

Background:

• Named Entity Recognition (NER) is crucial in medical NLP for identifying key clinical information.
• Traditional NER methods require extensive annotated data, posing resource challenges.
• Large Language Models (LLMs) offer innovative few-shot learning approaches for NER.

Purpose of the Study:

• To evaluate the impact of annotation guidelines within LLM prompts on few-shot NER performance.
• To assess this impact across diverse medical text corpora.

Main Methods:

• Eight prompt patterns were designed, combining few-shot examples with varying annotation guideline complexity.
• Performance was evaluated using three LLMs (GPT-4o, Claude 3.5 Sonnet, gpt-oss-120b) on three medical corpora (i2b2-2014, i2b2-2012, MedTxt-CR).
• Accuracy metrics included precision, recall, and F1 score, aligned with relevant shared tasks.

Main Results:

• The inclusion of detailed annotation guidelines in few-shot prompts generally led to improvements in recall and F1 scores.
• Specific prompt structures and guideline complexity influenced performance variations across LLMs and corpora.

Conclusions:

• Integrating annotation guidelines into LLM prompts is an effective strategy to boost NER performance, especially in few-shot scenarios.
• This method provides a practical and efficient way to develop accurate medical NLP systems, particularly in environments with limited annotated data.
• Annotation guidelines are vital for both evaluation and prompt engineering, optimizing LLM capabilities in specialized domains like medical NLP.