A framework for rigorous evaluation of human performance in human and machine learning comparison studies.

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Human-Computer Interaction

Background:

• Direct comparisons between human and machine learning (ML) performance are crucial for validating AI.
• The ML community lacks a standardized framework for evaluating human performance in comparative studies.
• Existing evaluation methods often overlook fundamental differences between human and algorithmic cognition.

Purpose of the Study:

• To address the lack of a standardized framework for human performance evaluation in ML comparisons.
• To propose guiding principles for designing robust human evaluation studies.
• To enhance the accuracy and reproducibility of human-AI performance comparisons.

Main Methods:

• Demonstration of common pitfalls in human performance evaluation design.
• Proposal of a standardized framework with three key principles.
• Illustration of the framework's application using a one-shot learning task study.

Main Results:

• Identified critical considerations for designing human evaluations, including understanding cognitive differences.
• Emphasized trial matching between human participants and algorithms.
• Advocated for adopting best practices from psychology research, including supplementary data collection and ethical protocols.

Conclusions:

• The proposed framework offers a standardized approach for evaluating human performance against ML algorithms.
• Adoption of this framework can improve the reliability and reproducibility of AI performance comparisons.
• This standardization is vital for advancing the field of artificial intelligence.