Building an Evaluation Scale using Item Response Theory.

Area of Science:

• Natural Language Processing (NLP)
• Psychometrics
• Computational Linguistics

Background:

• Standard evaluation of NLP methods relies on gold-standard test sets and metrics like accuracy, precision, recall, and F1.
• Current NLP evaluations assume all test items possess equal difficulty and discriminating power, which is a flawed assumption.

Purpose of the Study:

• To propose and demonstrate Item Response Theory (IRT) as an alternative for gold-standard test-set generation and NLP system evaluation.
• To leverage IRT's ability to characterize individual item difficulty and discriminating power for more nuanced NLP assessment.

Main Methods:

• Applied Item Response Theory (IRT) from psychometrics to NLP.
• Generated a gold-standard test set for the Recognizing Textual Entailment task.
• Collected a large dataset of human responses to the test set.
• Fitted an IRT model to the human response data.

Main Results:

• The IRT model enables a comparison of NLP systems against human performance benchmarks.
• IRT provides deeper insights into NLP system performance beyond traditional metrics.
• Demonstrated that high accuracy does not always correlate with high IRT scores, highlighting the influence of item characteristics.

Conclusions:

• Item Response Theory (IRT) offers a more sophisticated framework for evaluating NLP systems and test sets.
• IRT accounts for item difficulty and discrimination, leading to more accurate assessments of system capabilities.
• This psychometric approach enhances the reliability and interpretability of NLP evaluation results.