A scalable framework for benchmark embedding models in semantic health-care tasks.

Area of Science:

• Natural Language Processing
• Biomedical Informatics
• Machine Learning

Background:

• Text embeddings are crucial for semantic tasks like retrieval augmented generation (RAG).
• Healthcare applications of text embeddings are limited by a lack of standardized benchmarking.
• Existing methods are insufficient for evaluating embedding models in complex healthcare scenarios.

Purpose of the Study:

• To introduce a scalable benchmarking method for evaluating text embedding models in healthcare.
• To assess the performance of various embedding models across diverse medical semantic tasks.
• To provide a framework for selecting optimal embedding models for healthcare applications.

Main Methods:

• Evaluated 39 embedding models on 7 medical semantic similarity tasks.
• Utilized diverse datasets including patient data (Mount Sinai, MIMIC IV), PubMed articles, and synthetic data.
• Assessed semantic textual similarity (STS) using Spearman rank correlation and reframed tasks as retrieval problems evaluated by mean reciprocal rank and recall at k.

Main Results:

• Larger embedding models (>7b parameters) generally outperformed smaller models, especially in long-context tasks.
• NV-Embed-v1 (7b) excelled in short tasks but underperformed in long-context scenarios.
• Smaller models (e.g., b1ade-embed, 335M) matched larger models in short tasks, while larger models significantly led in long retrieval tasks.

Conclusions:

• The developed benchmarking framework is scalable and flexible for healthcare semantic tasks.
• This structured approach guides the selection of appropriate embedding models for diverse healthcare applications.
• Effective model selection enhances critical applications such as semantic search and retrieval-augmented generation (RAG).