Generalizable and scalable multistage biomedical concept normalization leveraging large language models.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Computational Linguistics

Background:

• Biomedical entity normalization is crucial for analyzing unstructured clinical data.
• Large Language Models (LLMs) show promise in Natural Language Processing (NLP) but are understudied for normalization tasks.

Purpose of the Study:

• To evaluate the effectiveness of proprietary and open-source LLMs in improving biomedical entity normalization.
• To assess LLM integration with existing rule-based normalization systems.

Main Methods:

• A two-step LLM integration approach was used: generating alternative phrasings and pruning candidate Unified Medical Language System (UMLS) concepts.
• Proprietary (GPT-3.5-turbo) and open-source (Vicuna) LLMs were combined with rule-based systems (MetaMapLite, QuickUMLS, BM25) and OpenAI embeddings.
• Performance was measured using $F_{\beta }$ (favoring recall) and F1 scores.

Main Results:

• GPT-3.5-turbo improved $F_{\beta }$ and F1 scores by up to +16.5 and +16.2.
• The open-source Vicuna model yielded greater improvements, increasing scores by up to +20.2 and +21.7.
• LLM integration enhanced performance across various normalization systems.

Conclusions:

• General-purpose LLMs, both proprietary and open-source, can significantly improve biomedical entity normalization performance.
• Existing normalization tools can be enhanced by LLMs without the need for fine-tuning.