Machine Translation Optimization

Home
Papers & Talks
Projects
Mentoring
CV

Machine Translation Optimization

Description

In this project, we looked at different aspects of MT Parameter optimization. More specifically how the choice of an optimizer and optimization metric can influence the end-to-end performance.

Related Publications

Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length
Francisco Guzmán, Preslav Nakov, Stephan Vogel. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning (CoNLL), pages 62-72, 2015.
PDF Abstract

Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length

We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.

BibTex

Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length


                @InProceedings{guzman-nakov-vogel:2015:CoNLL, author    = {Guzm\'{a}n, Francisco  and  Nakov, Preslav  and  Vogel, Stephan}, title     = {Analyzing Optimization for Statistical Machine Translation: MERT Learns Verbosity, PRO Learns Length}, booktitle = {Proceedings of the Nineteenth Conference on Computational Natural Language Learning}, month     = {July}, year      = {2015}, address   = {Beijing, China}, publisher = {Association for Computational Linguistics}, pages     = {62--72}, url       = {http://www.aclweb.org/anthology/K15-1007}, abstract = {We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets.
  Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.}
  }

Slides

Parameter Optimization for Statistical Machine Translation: It Pays to Learn from Hard Examples
Preslav Nakov, Fahad Al Obaidli, Francisco Guzmán, and Stephan Vogel. In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP'13) 2013.
PDF Abstract

Parameter Optimization for Statistical Machine Translation: It Pays to Learn from Hard Examples

Research on statistical machine translation has focused on particular translation directions, typically with English as the target language, e.g., from Arabic to English. When we reverse the translation direction, the multiple reference translations turn into multiple possible inputs, which offers both challenges and opportunities. We propose and evaluate several strategies for making use of these multiple inputs: (a) select one of the datasets, (b) select the best input for each sentence, and (c) synthesize an input for each sentence by fusing the available inputs. Surprisingly, we find out that it is best to tune on the hardest available input, not on the one that yields the highest BLEU score. This finding has implications on how to pick good translators and how to select useful data for parameter optimization in SMT.

BibTex

Parameter Optimization for Statistical Machine Translation: It Pays to Learn from Hard Examples


                @inproceedings{nakov:2013:parameter,
 abstract = {Research on statistical machine translation has focused on particular translation directions, typically with English as the target language, e.g., from Arabic to English. When we reverse the translation direction, the multiple reference translations turn into multiple possible inputs, which offers both challenges and opportunities. We propose and evaluate several strategies for making use of these multiple inputs: (a) select one of the datasets, (b) select the best input for each sentence, and (c) synthesize an input for each sentence by fusing the available inputs. Surprisingly, we find out that it is best to tune on the hardest available input, not on the one that yields the highest BLEU score. This finding has implications on how to pick good translators and how to select useful data for parameter optimization in SMT.},
 author = {Nakov, Preslav and Al Obaidli, Fahad and Guzm{\'a}n, Francisco and Vogel, Stephan},
 booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing ({RANLP}'13)},
 month = {September},
 title = {Parameter Optimization for Statistical Machine Translation: It Pays to Learn from Hard Examples},
 year = {2013}
}

Slides

A Tale about PRO and Monsters
Preslav Nakov, Francisco Guzmán, and Stephan Vogel. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL'13), pages 12-17, 2013.
PDF Abstract

A Tale about PRO and Monsters

While experimenting with tuning on long sentences, we made an unexpected discovery: that PRO falls victim to monsters -overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning and can cause testing BLEU to drop by several points absolute. We propose several effective ways to address the problem, using length- and BLEU+1- based cut-offs, outlier filters, stochastic sampling, and random acceptance. The best of these fixes not only slay and protect against monsters, but also yield higher stability for PRO as well as improved test-time BLEU scores. Thus, we recommend them to anybody using PRO, monster-believer or not.

BibTex

A Tale about PRO and Monsters


                @inproceedings{nakov-guzman-vogel:2013:Short,
 abstract = {While experimenting with tuning on long sentences, we made an unexpected discovery: that PRO falls victim to monsters -overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning and can cause testing BLEU to drop by several points absolute. We propose several effective ways to address the problem, using length- and BLEU+1- based cut-offs, outlier filters, stochastic sampling, and random acceptance. The best of these fixes not only slay and protect against monsters, but also yield higher stability for PRO as well as improved test-time BLEU scores. Thus, we recommend them to anybody using PRO, monster-believer or not.},
 address = {Sofia, Bulgaria},
 author = {Nakov, Preslav  and  Guzm{\'a}n, Francisco  and  Vogel, Stephan},
 booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics ({ACL'13})},
 link = {http://www.aclweb.org/anthology/P13-2003},
 month = {August},
 pages = {12--17},
 title = {A Tale about {PRO} and Monsters},
 year = {2013}
}

Slides

Optimizing for Sentence-Level BLEU+1 Yields Short Translations
Preslav Nakov, Francisco Guzmán, and Stephan Vogel. In Proceedings of the 24rd International Conference on Computational Linguistics (COLING 2012), pages 1979–1994, 2012.
PDF Abstract

Optimizing for Sentence-Level BLEU+1 Yields Short Translations

We study a problem with pairwise ranking optimization (PRO): that it tends to yield too short translations. We find that this is partially due to the inadequate smoothing in PRO’s BLEU+1, which boosts the precision component of BLEU but leaves the brevity penalty unchanged, thus destroying the balance between the two, compared to BLEU. It is also partially due to PRO optimizing for a sentence-level score without a global view on the overall length, which introducing a bias towards short translations; we show that letting PRO optimize a corpus-level BLEU yields a perfect length. Finally, we find some residual bias due to the interaction of PRO with BLEU+1: such a bias does not exist for a version of MIRA with sentence-level BLEU+1. We propose several ways to fix the length problem of PRO, including smoothing the brevity penalty, scaling the effective reference length, grounding the precision component, and unclipping the brevity penalty, which yield sizable improvements in test BLEU on two Arabic-English datasets: IWSLT (+0.65) and NIST (+0.37).

BibTex

Optimizing for Sentence-Level BLEU+1 Yields Short Translations


                @inproceedings{nakov-guzman-vogel:2012:PAPERS,
  author    = {Nakov, Preslav  and  Guzm{\'a}n, Francisco  and  Vogel, Stephan},
  title     = {Optimizing for Sentence-Level {BLEU}+1 Yields Short Translations},
  booktitle = {Proceedings of the 24rd International Conference on Computational Linguistics (COLING) 2012 },
  month     = {December},
  year      = {2012},
  address   = {Mumbai, India},
  publisher = {The {COLING} 2012 Organizing Committee},
  pages     = {1979--1994},
  url       = {http://www.aclweb.org/anthology/C12-1121},
  abstract = {We study a problem with pairwise ranking optimization (PRO): that it tends to yield too short translations. We find that this is partially due to the inadequate smoothing in PRO’s BLEU+1, which boosts the precision component of BLEU but leaves the brevity penalty unchanged, thus destroying the balance between the two, compared to BLEU. It is also partially due to PRO optimizing for a sentence-level score without a global view on the overall length, which introducing a bias towards short translations; we show that letting PRO optimize a corpus-level BLEU yields a perfect length. Finally, we find some residual bias due to the interaction of PRO with BLEU+1: such a bias does not exist for a version of MIRA with sentence-level BLEU+1. We propose several ways to fix the length problem of PRO, including smoothing the brevity penalty, scaling the effective reference length, grounding the precision component, and unclipping the brevity penalty, which yield sizable improvements in test BLEU on two Arabic-English datasets: IWSLT (+0.65) and NIST (+0.37).}
    }

Slides

Last updated January 29, 2017.
Created with git, jekyll, bootstrap, and sublime text.
Website template available at github fork from Adam Lopez's site.