Comparative Evaluation of Pre-Trained Language Models for Biomedical Information Retrieval

Area of Science:

• Biomedical Informatics
• Information Retrieval
• Natural Language Processing

Background:

• Efficient information retrieval (IR) is crucial for navigating the vast biomedical literature.
• Pre-trained language models offer advanced vector representations for search queries and documents.
• Algorithms like Cross-Encoders, SentenceBERT, and ColBERT leverage these models for IR.

Purpose of the Study:

• To evaluate the effectiveness of Cross-Encoders, SentenceBERT, and ColBERT in estimating biomedical document relevance for search queries.
• To compare the performance of these vectorization algorithms using the OHSUMED dataset.
• To assess the impact of fine-tuning on algorithm performance.

Main Methods:

• Utilized the OHSUMED dataset for evaluating information retrieval algorithms.
• Employed Cross-Encoders, SentenceBERT, and ColBERT for generating vector representations.
• Compared algorithm-computed relevance scores against provided relevance labels using boxplots and Spearman's rank correlations.

Main Results:

• Sentence-BERT demonstrated moderate outperformance compared to Cross-Encoders and ColBERT.
• Additional fine-tuning on a subset of OHSUMED labels yielded negligible benefits for performance.
• The vectorization algorithms showed varying degrees of success in estimating relevance.

Conclusions:

• Sentence-BERT is a promising vectorization algorithm for biomedical information retrieval tasks.
• Current fine-tuning strategies offer limited advantages for these specific algorithms and dataset.
• Further research with larger, dedicated datasets is needed for systematic optimization and end-user evaluation.