EMNLP 2019 parallel iterative edit models for local sequence transduction広樹 本間
?
- The document presents a Parallel Iterative Edit (PIE) model for local sequence transduction tasks like grammatical error correction.
- The PIE model achieves accuracy competitive with encoder-decoder models by predicting edits instead of tokens, iteratively refining predictions, and factorizing logits over edits and tokens to leverage pre-trained language models.
- Experiments show the PIE model provides a 5-15x speed improvement over encoder-decoder models for grammatical error correction while maintaining comparable accuracy.
The document proposes the Levenshtein Transformer (LevT), a new sequence generation model that uses insertion and deletion operations rather than autoregressive generation. LevT achieves comparable or better results than Transformer baselines on machine translation and text summarization tasks, with up to 5x faster decoding speed. LevT formulates sequence generation and refinement as a Markov decision process and learns dual insertion and deletion policies through imitation learning. Experiments show LevT is effective for machine translation, text summarization, and automatic post-editing tasks.
This document introduces a method called "co-curricular learning" that dynamically combines clean-data selection and domain-data selection for neural machine translation. It applies an EM-style optimization procedure to refine the "co-curriculum." Experimental results on two domains demonstrate the effectiveness of the method and properties of the data scheduled by the co-curriculum.