Evaluation of Chunking and Embedding Strategies for Local Document Retrieval Using an Open-Source LLM in a Hospital.

Area of Science:

• Natural Language Processing
• Information Retrieval
• Artificial Intelligence

Background:

• Retrieval-Augmented Generation (RAG) enhances context-aware and traceable information access.
• Domain-specific RAG systems require tailored strategies for optimal performance.
• This study focuses on RAG for administrative documents at University Hospital Halle.

Purpose of the Study:

• To explore chunking and embedding strategies for a RAG-based question-answering system.
• To evaluate model selection, parameter tuning, and retrieval performance.
• To lay the groundwork for a RAG chatbot to aid hospital staff in accessing documents.

Main Methods:

• Preprocessed and chunked a corpus of 1,219 administrative documents.
• Evaluated eight embedding models using Similarity Score and Maximum Marginal Relevance (MMR) retrievers.
• Analyzed top models (Jinaai-v3, Aari1995) with varied parameters and ensemble retrievers.

Main Results:

• Aari1995 achieved a 92.3% Top10 score, showing stable performance.
• Jinaai-v3 excelled in Top5 (84.6%) and Top3 (76.9%) scores but was less stable.
• Ensemble retrievers improved quality; Jinaai-v3 offered faster vector store generation; Similarity Score outperformed MMR.

Conclusions:

• Chunking and embedding significantly impact RAG retrieval effectiveness.
• Jinaai-v3 and Aari1995 offer distinct trade-offs in stability, accuracy, and efficiency.
• Findings support a locally executable RAG system, guiding future parameter optimization.