Clinical note comparison and data retrieval via embedding vectors: model selection, metrics, and convergence.

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Medicine
• Machine Learning

Background:

• Embedding models are key to generative AI, converting text to numerical vectors for semantic representation.
• Their efficacy in clinical settings, particularly with diverse languages, requires thorough evaluation.
• This study assesses embedding models for clinical note analysis and patient record retrieval.

Purpose of the Study:

• To evaluate the performance of various embedding models in detecting semantic differences within clinical notes.
• To assess the utility of embedding models for retrieving specific patient data from medical records.
• To compare model sensitivity to text perturbations and their effectiveness across different languages and tasks.

Main Methods:

• Eight embedding models were tested on synthetic discharge summaries in English, Swedish, and Finnish.
• Semantic sensitivity was measured by analyzing vector distances after text modifications (deletion, modification, paraphrasing).
• Two top-performing and two lower-performing models were further evaluated on real patient data retrieval tasks.

Main Results:

• Embedding models effectively captured semantic changes, with deletions/modifications yielding greater vector distance than paraphrasing.
• Model performance varied significantly; Qwen3-Embedding-8B demonstrated superior accuracy in directional semantic change detection compared to multilingual-E5-large.
• Retrieval task performance showed model-dependent variations, with Qwen3-Embedding-8B excelling in diagnosis-related queries.

Conclusions:

• The selection of an embedding model critically impacts the successful transfer of clinically relevant information.
• Differences in model performance are substantial enough to affect end-user access to crucial data.
• Model limitations are context-dependent, highlighting the need for careful model selection in clinical AI applications.