RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) with external knowledge for factual accuracy.
• RAG system performance depends on complex interactions between modular components, creating a challenging configuration space for developers.
• Existing methods lack systematic approaches for comparing and diagnosing RAG configurations.

Purpose of the Study:

• To introduce RAGExplorer, a visual analytics system designed for systematic comparison and diagnosis of RAG configurations.
• To provide developers with a tool to navigate the opaque configuration space of RAG systems.
• To facilitate understanding of performance trade-offs associated with different RAG design choices.

Main Methods:

• Developed RAGExplorer, a visual analytics system with a macro-to-micro workflow.
• Enabled high-level survey of performance across numerous RAG configurations.
• Facilitated deep-dive analysis into failure cases, exploring the impact of retrieved information, and interactive hypothesis testing.

Main Results:

• RAGExplorer allows developers to survey the RAG performance landscape and identify effective design choices.
• The system enables detailed investigation of failure cases and the influence of retrieved context on LLM outputs.
• User studies and case studies validated RAGExplorer's effectiveness in empowering RAG system development.

Conclusions:

• RAGExplorer effectively addresses the challenge of navigating complex RAG design spaces.
• The visual analytics approach aids developers in optimizing RAG configurations for improved LLM performance.
• Systematic comparison and diagnosis are crucial for advancing RAG system development and deployment.