Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015
1 like77 views
This document discusses using neural reranking to improve the subjective quality of machine translation. It finds that reranking N-best lists generated by a baseline machine translation system using neural models leads to improvements in both automatic metrics like BLEU and manual evaluations of translation quality. A qualitative analysis shows that reranking most improves reordering, insertion, and conjugation errors while having less success with terminology. The analysis suggests neural reranking is an effective technique for machine translation enhancement.
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015
- 1. Neural Reranking Helps Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, Satoshi Nakamura Nara Institute of Science and Technology (NAIST), Japan Neural Reranking Quantitative Results Experimental Setup he has a cold Input T2S N-best w/MT Features 1. ˤϺ֤äƤ t=-0.5 l=-5.6 | -6.1 2. ˤLа֤äƤ t=-0.9 l=-5.8 | -6.7 3. ˤLа t=-1.5 l=-5.3 | -6.8 4. ˤLа t=-1.9 l=-5.4 | -7.3 Neural Model Neural Features nmt=-5.8 nmt=-5.5 nmt=-3.4 nmt=-5.2 2. ˤϺ֤äƤ t=-0.5 l=-5.6 nmt=-5.8 | -10.9 3. ˤLа֤äƤ t=-0.9 l=-5.8 nmt=-5.5 | -11.2 1. ˤLа t=-1.5 l=-5.3 nmt=-3.4 | -9.2 4. ˤLа t=-1.9 l=-5.4 nmt=-5.2 | -12.5 Rescored/Reranked N-best Reranking Data: ASPEC Scientific Abstracts C Japanese ? English, Chinese Baseline: NAIST WAT2014 Tree-to-String System C Strong baseline achieving high scores C Implemented using Travatar (http://phontron.com/travatar) Neural MT Model: Attentional model C Trained ~500k sent., 256 hidden nodes, 2 model ensemble C Use words occurring 3+ times (vocab 50,000~80,000) C Trained w/ lamtram (http://github.com/neubig/lamtram) Automatic Evaluation: BLEU, RIBES Manual Evaluation: WAT 2015 HUMAN Score Research Questions 1. Does reranking with neural MT models improve subjective impressions of translation results? 2. If so, what are the qualitative differences between reranked and non-reranked output? 3. How big of an n-best list do we need? en-ja ja-en zh-ja ja-zh 0 10 20 30 40 50 BLEU en-ja ja-en zh-ja ja-zh 70 72 74 76 78 80 82 84 86 Base Rerank RIBES +1.6 +2.8 +2.5 +1.5 +1.8 +2.7 +1.4 +1.8 Confirm what we know: Neural reranking helps BLEU. en-ja ja-en zh-ja ja-zh 0 10 20 30 40 50 60 70 Base Rerank HUMAN +12.5 +23.7 +10.0 +4.2 Show what we didn't know: Also helps manual eval. Reranking and N-best Size All data sets improve approximately log-linear Little saturation even at 100-best Detailed Analysis #1 Improvement: Phrasal Reordering (+26, -4) Source Base Rerank Ref ֢2ˤƤϡֱcθܞƤˌ뻯ѧФˡ kࡢӲYƤw褦 In case 2, reddening, induration, and skin ulcer appeared during chemical therapy for liver metastasis of rectal cancer. In case 2, occurred during chemotherapy for liver metastasis of rectal cancer, flare, induration, skin ulcer. In case 2, the flare, induration, skin ulcer was produced during the chemotherapy for hepatic metastasis of rectal cancer. #2 Improvement: Auxiliary Verb Ins./Del. (+15, -0) Source Base Rerank Ref ˤä֧䷽ʽϱΤ褦ʤ ˤä?롣 Governing equation derived by this method is useful for turbulent shear flow like turbulent flow near wall. The governing equation is obtained by this is also useful for such as wall turbulence shear flow. The governing equation obtained by this is also useful for shear flow such as wall turbulence. #3 Improvement: Coordinate Structures (+13, -2) Source Base Rerank Ref ``ӹϸܶȹˤĤʼӟȥ֥` ˤФ Laser work is done by local heating and ablation with high density light flux. The laser processing is carried out by local heating by high- density luminous flux and ablation. The laser processing is carried out by local heating and ablation by high-density flux. #4 Improvement: Verb Agreement (+6, -0) Source Base Rerank Ref ߥ奢\֥åȷӻˤⴥ줿 Langmuir-Blodgett method and inclusion compounds are mentioned. Langmuir-Blodgett method and inclusion is also discussed. Langmuir-Blodgett method and inclusion are also mentioned. Type Improved Degraded % Impr. Reordering 55 9 86% Deletion 20 10 67% Insertion 19 2 90% Substitution 15 11 58% Conjugation 8 1 89% Total 117 33 78% Overall improvements re-confirmed Particularly reordering, insertion, and conjugation errors Qualitative Analysis What Didn't Work: Terminology (+2, -4) Source Base Rerank Ref ä⾀ӋyäƤ롣 Infrared ray applied measurement using radiant heat is useful for stress analysis. The infrared application measurement using radiant heat is useful in the stress analysis. Infrared ray application measurement using radiation heat is useful for stress analysis. Conclusion Reranking with neural MT models leads to subjective improvements in MT quality Future work includes comparisons with neural language models or neural MT w/o reranking