Scaling Responsible Medical Text Retrieval Across Silos: Evaluating Small Language Models with Retrieval-Augmented...

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Natural Language Processing

Background:

• Efficient medical text retrieval is crucial for managing Electronic Medical Records (EMR).
• Small Language Models (SLMs) offer potential for specialized tasks but require optimization for performance and scalability.
• Retrieval Augmented Generation (RAG) frameworks enhance SLM capabilities by integrating external knowledge.

Purpose of the Study:

• To evaluate the effectiveness of Small Language Models (SLMs) combined with FAISS (Facebook AI Similarity Search) and Dynamic Top-k for diabetes-focused medical text retrieval.
• To assess the performance and scalability of lightweight SLM-RAG pipelines in cloud-based EMR environments.
• To compare the accuracy, precision, and latency of different SLM configurations.

Main Methods:

• Evaluation of the all-mpnet-base-v2 model for medical text retrieval.
• Implementation and testing of FAISS for efficient similarity search.
• Application of Dynamic Top-k sampling to optimize retrieval results.
• Benchmarking latency and accuracy metrics at increasing data scales.

Main Results:

• The all-mpnet-base-v2 model demonstrated superior performance among the evaluated SLMs.
• FAISS maintained sub-2 millisecond latency even with a tenfold increase in data scale.
• Dynamic Top-k significantly improved precision and Normalized Discounted Cumulative Gain (nDCG).
• Lightweight SLM-RAG pipelines achieved accuracy comparable to larger models.

Conclusions:

• Lightweight SLM-RAG pipelines provide a highly scalable and accurate solution for diabetes-focused medical text retrieval.
• FAISS and Dynamic Top-k are effective components for optimizing SLM-RAG performance in terms of speed and result quality.
• These findings support the use of SLM-RAG for cloud-based EMR systems, balancing performance with resource efficiency.