Using tournaments to calculate AUROC for zero-shot classification with LLMs

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Large language models (LLMs) show promise in zero-shot classification but lack a fair comparison method due to their static decision boundaries.
• Existing methods struggle to directly compare LLMs with supervised classifiers.

Purpose of the Study:

• To propose and evaluate a novel method for fairly comparing LLMs on binary classification tasks.
• To transform classification into a pairwise comparison task solvable by LLMs.
• To leverage the Elo rating system for instance scoring and confidence ordering.

Main Methods:

• Binary classification tasks were reframed as pairwise comparisons between dataset instances.
• LLMs were employed to generate relative rankings of instances.
• The Elo rating system was utilized to score instances based on repeated pairwise comparisons.
• Scheduling algorithms were evaluated for comparison minimization.

Main Results:

• The proposed pairwise comparison method, utilizing LLMs and Elo ratings, demonstrated improved classification performance.
• The method provides a confidence ordering over dataset instances.
• The evaluation of scheduling algorithms showed effectiveness in minimizing comparisons.
• The approach offers more comprehensive information compared to traditional zero-shot classification.

Conclusions:

• The pairwise comparison method offers a robust and fair way to evaluate LLMs in classification.
• LLM-driven pairwise comparisons with Elo ratings enhance classification accuracy and provide valuable instance-level insights.
• This method advances the field of LLM evaluation and zero-shot learning.