How different is different? Systematically identifying distribution shifts and their impacts in NER datasets.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning

Background:

• Distribution shift is a known challenge in NLP, where models trained on one data domain perform poorly on another.
• Existing research lacks systematic studies on detecting and quantifying the impact of distribution shifts on NLP task performance.

Purpose of the Study:

• To systematically detect and measure two types of distribution shift: input shift and label shift.
• To investigate the impact of these shifts on model performance across various NLP tasks and datasets.
• To provide insights into NLP model pipeline definition and fine-tuning requirements.

Main Methods:

• Detection and measurement of input and label shifts.
• Evaluation across three distinct data representations.
• Analysis on 12 benchmark Named Entity Recognition (NER) datasets.

Main Results:

• Both input and label shifts cause significant performance degradation in NLP models.
• A specific example shows a 63-point F1 score drop when fine-tuning on a broad dataset and testing on a narrower one.
• Shift measurement correlates with the amount of data needed for effective fine-tuning.

Conclusions:

• Quantifying distribution shift is crucial for understanding NLP model reliability.
• Shift measurement can inform decisions on whether a model is usable off-the-shelf or requires fine-tuning.
• Measuring distribution shift is vital for defining robust NLP model pipelines.