A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Non-autoregressive (NAR) generation models accelerate inference in tasks like neural machine translation (NMT).
• NAR models achieve speedups at the expense of reduced accuracy compared to autoregressive (AR) models.
• Bridging the accuracy gap in NAR generation is a key research challenge.

Purpose of the Study:

• To systematically survey and compare various non-autoregressive translation (NAT) models.
• To categorize advancements in NAT across data manipulation, modeling, training, and decoding.
• To explore future research directions and applications of NAR models.

Main Methods:

• Categorization of NAT efforts into data manipulation, modeling, training criteria, and decoding algorithms.
• Review of techniques to improve NAR model accuracy.
• Discussion of benefits from pre-trained models for NAR generation.

Main Results:

• NAT models have seen significant development across multiple research fronts.
• Various strategies exist to mitigate the accuracy loss in NAR generation.
• NAR models show promise beyond machine translation in diverse NLP tasks.

Conclusions:

• This survey provides a comprehensive overview of the current state of NAR generation.
• It highlights key challenges and opportunities for future research in improving NAR model performance and applicability.
• The findings aim to guide researchers and practitioners in selecting and developing NAR solutions.