Benchmarking MeSH-augmented embeddings for biomedical document similarity

Area of Science:

• Biomedical Informatics
• Information Retrieval
• Natural Language Processing

Background:

• Efficient retrieval of biomedical literature is crucial due to its vast volume.
• Embedding-based methods enhance document retrieval over traditional keyword approaches.
• Integrating domain-specific terminologies like Medical Subject Headings (MeSH) into embedding models is underexplored.

Purpose of the Study:

• To compare hybrid methods integrating MeSH annotations with document embeddings.
• To benchmark these hybrid methods against traditional and transformer-based models.
• To evaluate the impact of MeSH integration on biomedical document retrieval performance.

Main Methods:

• Three hybrid methods (pre-annotation, post-annotation, post-reduction) integrating MeSH with document embeddings were developed.
• Methods were benchmarked against TF-IDF, Word2Vec, fastText, Doc2Vec, BioBERT, SciBERT, SPECTER, and SapBERT.
• Experiments utilized the RELISH corpus with cosine similarity and Word Mover's Distance (WMD) for evaluation.

Main Results:

• Fine-tuned BioBERT and SciBERT models demonstrated the best alignment with expert judgments.
• Doc2Vec and MeSH-based hybrid methods performed well, highlighting the value of combining vocabularies with embeddings.
• While MeSH integration offered modest improvements (2-4%), fine-tuned large models achieved up to 90% precision.

Conclusions:

• Performance gains from concept integration may be moderate, but the study successfully benchmarks structured vocabularies with embedding methods.
• These techniques are applicable for aligning literature with other data sources using controlled vocabularies.
• The Dockerized pipeline ensures reproducibility and supports future research in biomedical document retrieval.