Sequence to Sequence Learning with Neural Networks
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences.
Also cited · not yet reviewed (1)
- Bahdanau attention2014 · cited 4דWe were initially convinced that the LSTM would fail on long sentences due to its limited memory, and other researchers reported poor performance on long sentences with a model similar to ours [5, 2, 26].”From this paper · §Conclusion
Led to
- Bahdanau attention2014 · cited 10×, 7 in Method“Sutskever et al. 2014 reported that the neural machine translation based on RNNs with long short-term memory (LSTM) units achieves close to the state-of-the-art performance of the conventional phrase-based machine transl…”From Bahdanau attention · §Background: Neural Machine Translation
- Ask Your Neurons2015 · cited 3×, 1 in Method“LSTM has been recently shown to be effective in learning a variable-length sequence-to-sequence mapping [5, 28].”From Ask Your Neurons · §Approach
- GNMT2016 · cited 6×, 2 in Method“This observation is similar to previous observations that deep LSTMs significantly outperform shallow LSTMs [41].”From GNMT · §Model Architecture
- ByteNet2016 · cited 5×, 2 in Method“The first mechanism involves the stacking of the decoder on top of the representation of the encoder in a manner that preserves the temporal resolution of the sequences; this is in contrast with architectures that encode…”From ByteNet · §Introduction
- ConvS2S2017 · cited 3דSequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al., 2015; Nallapati et al.…”From ConvS2S · §Introduction
- Transformer2017 · cited 2×, 1 in Method“Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].”From Transformer · §Model Architecture
- T52019 · cited 2×, 1 in Method“The original Transformer consisted of an encoder-decoder architecture and was intended for sequence-to-sequence (Sutskever et al. 2014; Kalchbrenner et al. 2014) tasks.”From T5 · §Setup
Abstract
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.