LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Data Visualization

Background:

• Evaluating large language models (LLMs) using automatic side-by-side comparisons (LLM-as-a-judge) is promising but faces scalability and interpretability issues.
• Current methods hinder in-depth analysis of model performance and understanding the reasons behind differing LLM outputs.

Purpose of the Study:

• To introduce LLM Comparator, a novel visual analytics tool designed to address the challenges in analyzing side-by-side LLM evaluations.
• To provide analytical workflows enabling users to understand when and why LLMs differ in performance and response generation.

Main Methods:

• Iterative design and development of LLM Comparator through collaboration with LLM practitioners at Google.
• Incorporation of visual analytics techniques to facilitate both in-depth analysis of individual examples and overview of large datasets.
• User-centered qualitative feedback collection and integration into the tool's refinement process.

Main Results:

• LLM Comparator facilitates in-depth analysis of individual LLM-generated examples.
• The tool enables users to visually overview and flexibly slice evaluation data, aiding pattern identification.
• Qualitative feedback confirms the tool's utility in formulating hypotheses and gaining insights for LLM improvement.

Conclusions:

• LLM Comparator enhances the scalability and interpretability of LLM evaluations.
• The tool empowers researchers and developers to gain deeper insights into LLM behavior and drive model improvements.
• LLM Comparator has been integrated into Google's LLM evaluation platforms and made open-source for broader adoption.