LLM-as-a-Judge: automated evaluation of search query parsing using large language models

Area of Science:

• Natural Language Processing
• Information Retrieval
• Machine Learning Evaluation

Background:

• Traditional evaluation methods for search systems are insufficient for Large Language Models (LLMs).
• New methodologies are required to assess the structured outputs generated by LLMs in search applications.

Purpose of the Study:

• To propose a general framework for evaluating structured outputs using LLMs.
• To focus on search query parsing within an online classified platform as a use case.
• To introduce novel LLM-based evaluation techniques for improved reliability.

Main Methods:

• Leveraging LLMs' contextual reasoning for evaluation.
• Implementing three methodologies: Pointwise, Pairwise, and Pass/Fail assessments.
• Introducing a Contextual Evaluation Prompt Routing strategy to mitigate hallucinations.

Main Results:

• LLM-based evaluation achieved approximately 90% agreement with human judgments.
• Experiments were conducted on both small- and large-scale datasets.
• The proposed framework demonstrated effectiveness in query parsing.

Conclusions:

• LLM-driven evaluation is a scalable, interpretable, and effective alternative to traditional methods.
• The framework provides robust query parsing for real-world search systems.
• This approach enhances the reliability of LLM applications in search.