Comparing a Large Language Model with Previous Deep Learning Models on Named Entity Recognition of Adverse Drug...

Area of Science:

• Computational linguistics
• Biomedical informatics
• Artificial intelligence

Background:

• Deep learning models significantly enhance named entity recognition (NER) performance when fine-tuned on large datasets.
• Large language models (LLMs) offer few-shot learning capabilities for new tasks via in-context learning with prompts.
• Few-shot learning requires minimal examples, making it suitable for data-scarce scenarios.

Purpose of the Study:

• To compare the performance of state-of-the-art deep learning models fine-tuned on PubMed abstracts against LLMs using few-shot learning for adverse drug event (ADE) NER.
• To evaluate the efficacy of different machine learning approaches in identifying adverse drug events from biomedical text.

Main Methods:

• Fine-tuning a state-of-the-art deep learning model (Hussain et al.) on a large corpus of PubMed abstracts for ADE NER.
• Utilizing a large language model (ChatGPT-3.5) with few-shot learning for ADE NER, providing minimal examples and instructions.
• Direct comparison of the F1-Scores achieved by both methodologies.

Main Results:

• The fine-tuned deep learning model achieved a significantly higher F1-Score (97.6%) compared to the ChatGPT-3.5 model using few-shot learning (86.0%) for ADE NER.
• Fine-tuning on extensive data yielded superior performance over few-shot learning in this specific NER task.
• Few-shot learning demonstrated convenience but lower accuracy for ADE identification.

Conclusions:

• Fine-tuning deep learning models with substantial data remains the superior approach for high-performance adverse drug event named entity recognition.
• Few-shot learning with LLMs is a practical alternative when training data is limited, though performance is currently inferior.
• Future research should explore advanced prompting techniques with newer LLMs like GPT-4 and investigate fine-tuning strategies for LLMs like GPT-3.5.