Improving large language models for clinical named entity recognition via prompt engineering.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Biomedical Informatics

Background:

• Large language models (LLMs) demonstrate potential for processing complex clinical data.
• Clinical Named Entity Recognition (NER) is crucial for extracting information from electronic health records.
• Current methods often require extensive annotated datasets, limiting scalability.

Purpose of the Study:

• To evaluate the performance of GPT-3.5 and GPT-4 on clinical NER tasks.
• To develop and assess a task-specific prompt framework to enhance LLM capabilities in clinical settings.
• To compare LLM performance against established models like BioClinicalBERT.

Main Methods:

• Two clinical NER tasks were performed: concept extraction from MTSamples and adverse event identification from VAERS.
• A prompt framework was developed, including baseline, guideline-based, error analysis-based, and few-shot learning prompts.
• Model performance was evaluated using relaxed F1 scores and compared to BioClinicalBERT.

Main Results:

• GPT-3.5 and GPT-4 performance improved significantly with the task-specific prompt framework.
• With all prompt components, GPT-4 achieved F1 scores of 0.861 (MTSamples) and 0.736 (VAERS).
• While LLM performance did not surpass BioClinicalBERT, it showed promising results with minimal training data.

Conclusions:

• Task-specific prompts enhance the feasibility of LLMs for clinical NER.
• LLMs, particularly GPT-4, show potential to approach state-of-the-art performance with careful prompt engineering.
• Further research and refined evaluation schemas are needed to fully leverage LLMs in clinical applications.