Ask me in your own words: paraphrasing for multitask question answering

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Multitask learning has advanced Natural Language Processing (NLP), exemplified by the decaNLP benchmark.
• The decaNLP benchmark unifies 10 NLP tasks within a single model, often using question answering as a framework.

Purpose of the Study:

• To investigate the robustness of models trained on decaNLP when faced with paraphrased questions.
• To develop and evaluate methods for improving model resilience to linguistic variations in natural language understanding tasks.

Main Methods:

• A crowd-sourced corpus of paraphrased questions (PQ-decaNLP) was created, annotated with specific paraphrase phenomena.
• MQAN and T5 models were retrained using the PQ-decaNLP corpus to assess performance improvements.
• Paraphrase detection and generation tasks were explored to understand knowledge transfer in multitask learning.

Main Results:

• Models trained on decaNLP exhibit significant performance degradation with simple question paraphrasing.
• Training with PQ-decaNLP enhances model robustness and, in some cases, improves performance on original questions.
• While models can learn paraphrase detection and generation, this knowledge does not effectively transfer to other decaNLP tasks.

Conclusions:

• Robustness to paraphrasing is crucial for practical NLP applications beyond benchmark performance.
• Targeted training data, like PQ-decaNLP, can significantly improve model generalization.
• Current multitask learning setups do not inherently transfer paraphrasing knowledge effectively across diverse NLP tasks.