Comparative Analysis of Large Language Models in Chinese Medical Named Entity Recognition.

Area of Science:

• Natural Language Processing
• Biomedical Informatics
• Artificial Intelligence

Background:

• Large language models (LLMs) excel in general named entity recognition (NER).
• Effectiveness of LLMs for Chinese biomedical named entity recognition (BNER) is underexplored.
• Chinese electronic medical records (EMR) present unique challenges for BNER.

Purpose of the Study:

• Evaluate typical LLMs (ChatGLM2-6B, GLM-130B, GPT-3.5, GPT-4) on Chinese BNER.
• Assess LLM performance with zero-shot and few-shot prompting.
• Investigate the impact of instruction fine-tuning on Chinese BNER.

Main Methods:

• Utilized real-world Chinese EMR and public datasets (CCKS2017).
• Evaluated LLMs using zero-shot, few-shot, and instruction fine-tuning approaches.
• Compared LLM performance against task-specific models like BiLSTM+CRF and DGAN.

Main Results:

• LLMs show promising but limited performance with basic prompting.
• Instruction fine-tuning substantially boosts LLM effectiveness for Chinese BNER.
• Fine-tuned ChatGLM2-6B surpassed BiLSTM+CRF on real-world data.
• Fine-tuned GPT-3.5 excelled on CCKS2017 but lagged behind state-of-the-art DGAN.

Conclusions:

• This study is the first to evaluate LLMs for Chinese BNER.
• LLMs, especially after fine-tuning, offer significant potential for Chinese BNER.
• Actionable guidelines are provided for leveraging LLMs in this domain.