Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Natural Language Processing

Background:

• Evaluating AI-powered Question-Answering (Q&A) systems often relies on expert ratings.
• The cost and scalability of expert evaluations can be a bottleneck in AI development.
• Exploring alternative raters like laypeople or AI is crucial for efficient system assessment.

Purpose of the Study:

• To introduce a straightforward methodology for evaluating the adequacy of laypeople or AI in substituting for expert ratings of AI Q&A systems.
• To establish a benchmark for expert agreement and compare it against non-expert rater performance.
• To provide a transparent and structured method for assessing rater reliability.

Main Methods:

• Utilized weighted Cohen's Kappa to quantify inter-rater reliability.
• Established an expert agreement benchmark for comparison.
• Employed an inter-rater reliability matrix for visualizing and analyzing results.

Main Results:

• Findings indicate that laypeople and AI can, in specific contexts, achieve agreement levels comparable to or exceeding those of human experts.
• The effectiveness of non-expert raters was particularly notable when risk aversion was a consideration.
• The proposed approach demonstrated transparency and structure in assessing rater adequacy.

Conclusions:

• The developed approach offers a viable and adaptable method for assessing the suitability of non-expert raters in AI Q&A system evaluations.
• Layperson and AI raters show potential to augment or replace expert evaluations, especially in risk-sensitive applications.
• The methodology's flexibility allows for application across diverse AI evaluation contexts and rating criteria.