A survey on LLM-as-a-judge

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Accurate evaluation is vital but challenged by subjectivity and scale.
• Large Language Models (LLMs) show promise for automated evaluation.
• Ensuring the reliability of LLM-as-a-judge systems is a key challenge.

Purpose of the Study:

• To provide a comprehensive survey of LLM-as-a-judge systems.
• To address the core question of building reliable LLM-as-a-judge systems.
• To offer a unified framework and research agenda for trustworthy AI evaluation.

Main Methods:

• Formal definition and detailed classification of LLM-as-a-judge.
• Exploration of strategies to enhance LLM evaluation reliability (consistency, bias mitigation).
• Development of methodologies and a novel benchmark for evaluating LLM reliability.

Main Results:

• A structured survey and unified framework for LLM-as-a-judge.
• Proposed strategies and methodologies for improving LLM evaluation reliability.
• A novel benchmark for assessing the trustworthiness of LLM-based evaluations.

Conclusions:

• LLM-as-a-judge offers a scalable alternative to traditional evaluations.
• Careful design and standardization are crucial for reliable LLM-based assessment.
• This work provides theoretical foundations and practical guidance for developing trustworthy LLM evaluators.