Comparative analysis of generative LLMs for labeling entities in clinical notes

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Artificial Intelligence

Background:

• Large language models (LLMs) are increasingly applied to specialized domains.
• Zero-shot named entity recognition (NER) aims to identify entities without domain-specific training data.
• Clinical text poses unique challenges for automated information extraction.

Purpose of the Study:

• To evaluate and compare fine-tuned variations of generative LLMs for zero-shot clinical NER.
• To assess the performance of Llama 2 and Mistral models in identifying clinical entities.
• To determine the impact of different fine-tuning strategies (code, chat, instruction) on NER accuracy.

Main Methods:

• Utilized publicly available clinical case data with labeled diseases, symptoms, and procedures.
• Evaluated base, code-tuned, chat-tuned, and instruction-tuned Llama 2 and Mistral models.
• Assessed entity identification accuracy and the ability to retrieve entities in structured formats.

Main Results:

• Instruction fine-tuned models outperformed chat fine-tuned and base models in recognizing clinical entities.
• Models demonstrated improved performance when requested to provide simple output structures.
• The specific fine-tuning approach significantly impacts zero-shot NER capabilities in the clinical domain.

Conclusions:

• Instruction fine-tuning is a promising strategy for enhancing LLM performance in clinical NER.
• Simpler output formats facilitate more accurate entity extraction by LLMs.
• Further research can optimize LLM application for clinical information extraction.