Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds.

Area of Science:

• Natural Language Processing (NLP)
• Psychometrics
• Machine Learning

Background:

• Item Response Theory (IRT) traditionally relies on human response patterns (RP), which is a bottleneck for large datasets in deep neural network (DNN) training.
• Applying IRT to NLP tasks offers insights into model performance and behavior.

Purpose of the Study:

• To propose and validate a method for learning IRT models using DNN-generated RPs.
• To assess the effectiveness of this method for NLP tasks like natural language inference and sentiment analysis.
• To explore the utility of machine-generated IRT parameters for applications such as training set filtering.

Main Methods:

• Generating artificial response patterns (RPs) from DNN models.
• Learning IRT models using these machine-generated RPs.
• Quantitative and qualitative analyses comparing machine-learned IRT parameters with those from human RPs.
• Demonstrating a use-case for latent difficulty parameters in training set filtering.

Main Results:

• Parameters learned from human and machine RPs show medium to large positive correlations for NLP tasks.
• Using machine-estimated item difficulty for training set sampling outperforms baseline methods.
• Identified discrepancies between human expectations and machine-estimated item difficulty.

Conclusions:

• Learning IRT models with DNN-generated data is effective for NLP tasks.
• Machine-generated IRT parameters offer practical applications, such as optimized training set selection.
• This approach provides a scalable alternative to human data collection for IRT model learning in NLP.