Don't Use LLMs to Make Relevance Judgments.

Area of Science:

• Information Retrieval
• Natural Language Processing
• Artificial Intelligence

Background:

• Relevance judgments are crucial for evaluating information retrieval (IR) systems.
• Manual creation of relevance data is time-consuming and resource-intensive.
• Large language models (LLMs) offer potential for automating tasks in IR.

Purpose of the Study:

• To investigate the feasibility and implications of using LLMs to generate relevance judgments for IR evaluations.
• To determine if LLMs can serve as reliable proxies for human judges in assessing information relevance.
• To identify best practices for integrating LLMs into the relevance assessment process.

Main Methods:

• The study critically analyzes the concept of using LLMs for generating truth data in IR.
• It discusses the inherent limitations and potential biases introduced by LLM-generated judgments.
• It explores alternative, more appropriate methods for LLM integration in relevance assessment.

Main Results:

• Directly using LLMs to generate relevance judgments artificially caps the performance ceiling of evaluated IR systems.
• LLM-generated truth data can lead to flawed and unreliable IR evaluations.
• This approach undermines the purpose of rigorous system assessment.

Conclusions:

• LLMs should not be used to directly generate relevance judgments for IR evaluations.
• Employing LLMs as direct proxies for human judges in this capacity is detrimental to evaluation integrity.
• LLMs can be valuable tools in IR, but their application in generating ground truth requires careful consideration and alternative methodologies.