Benchmarking Transformer Embedding Models for Biomedical Terminology Standardization.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Machine Learning

Background:

• Biomedical databases suffer from inconsistent terminology, hindering machine learning and data integration.
• Standardizing terminology is crucial for effective use of biomedical data.

Purpose of the Study:

• To evaluate the effectiveness of transformer/large language models (LLMs) for standardizing biomedical terminology in the NIH Clinical Trials Registry (CTR).
• To benchmark LLM-based approaches against traditional text-matching algorithms using the World Health Organization Classification of Tumours (WHO System) as a gold standard.

Main Methods:

• Developed CANTOS (Clinical Trials Automated Nomenclature and Tumor Ontology Standardization) framework to extract and standardize tumor names from the CTR.
• Benchmarked 36 methods, including LLM/transformer text embeddings and traditional algorithms, against manually annotated WHO System terms.
• Assessed accuracy using a sample of 1,600 CTR tumor names.

Main Results:

• LLM/transformer-based embedding methods significantly outperformed text-matching approaches, achieving up to 69.4% accuracy.
• Text-matching methods achieved a maximum accuracy of 32.6%.
• A majority voting ensemble improved accuracy to 71.9%.

Conclusions:

• LLM/transformer embedding models are effective for standardizing biomedical terminology.
• The CANTOS framework provides a reproducible method for benchmarking machine learning in biomedical data standardization.
• Accurate terminology standardization enhances the utility of biomedical databases for research and machine learning.