Evaluating Large Language Models on Named Entity Recognition.

Area of Science:

• Artificial Intelligence
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in various tasks, but their evaluation in named entity recognition (NER) is underexplored.
• Existing LLM evaluations primarily focus on natural language understanding tasks like text classification and sentiment analysis.

Purpose of the Study:

• To comprehensively evaluate twenty-eight large language models (LLMs) on named entity recognition (NER) across diverse datasets and domains.
• To analyze LLM performance from four key perspectives: supervised fine-tuning (SFT), parameter scales, hallucination mitigation, and prompt design sensitivity.

Main Methods:

• Developed and utilized an LLM-based NER framework (LLM-NER) incorporating Recognition and Check phases to mitigate hallucinations.
• Evaluated 28 LLMs with parameter sizes ranging from 3 to 175 billion on 13 datasets across 5 domains.
• Analyzed LLM performance based on SFT, parameter scale, hallucination rates, and prompt variations.

Main Results:

• Supervised fine-tuning (SFT) enhances LLMs' ability to follow instructions for NER tasks.
• LLM performance generally correlates positively with increasing parameter scales.
• Hallucinations are present in all evaluated LLMs, but can be alleviated by implementing a checking mechanism.
• LLM performance is significantly influenced by prompt design.

Conclusions:

• LLM evaluation for NER requires specific frameworks that address challenges like hallucinations.
• Future research should focus on optimizing SFT, leveraging parameter scale, developing robust hallucination mitigation strategies, and refining prompt engineering for NER.
• The proposed evaluation methodology demonstrates high consistency with existing LLM evaluation leaderboards.