Leveraging large language models for rare disease named entity recognition.

Area of Science:

• Biomedical Natural Language Processing
• Artificial Intelligence in Healthcare

Background:

• Named Entity Recognition (NER) in rare diseases is challenging due to limited data and semantic ambiguity.
• Traditional supervised models struggle with the long-tail distributions common in rare disease data.

Purpose of the Study:

• To evaluate GPT-4o's performance for rare disease NER under low-resource conditions.
• To compare various prompt-based strategies, including zero-shot, few-shot learning, RAG, and fine-tuning.
• To introduce novel methods for semantically guided few-shot example selection.

Main Methods:

• Utilized GPT-4o with structured prompting incorporating domain knowledge and disambiguation rules.
• Implemented zero-shot, few-shot in-context learning, retrieval-augmented generation (RAG), and task-level fine-tuning.
• Developed two semantically guided few-shot example selection techniques to optimize performance and reduce labeling effort.

Main Results:

• GPT-4o achieved competitive or superior performance compared to BioClinicalBERT on the RareDis Corpus.
• Task-level fine-tuning demonstrated the strongest results, surpassing previous BioClinicalBERT baselines.
• Few-shot prompting offered excellent cost-performance, especially at lower token budgets, while RAG improved recall for specific entity types like signs and symptoms.

Conclusions:

• Prompt-optimized Large Language Models (LLMs) offer effective and scalable solutions for biomedical NER, particularly in data-scarce rare disease domains.
• GPT-4o presents a viable alternative to traditional supervised methods for rare disease NER.
• Further research into error analysis can guide post-processing and hybrid refinement strategies for improved accuracy.