Neural Machine Translation of Rare Words with Subword Units

ArticleRico Sennrich, Barry Haddow, Alexandra BirchResearch

Rico Sennrich, Barry Haddow, and Alexandra Birch present an approach enabling neural machine translation (NMT) systems to handle rare and out-of-vocabulary words using subword units. Instead of fixed vocabularies and dictionary back-offs, the technique employs segmentation algorithms like Byte Pair Encoding (BPE) for open-vocabulary translation.

Key points

  1. Traditional NMT architectures operate with fixed vocabularies and typically rely on back-off dictionaries for out-of-vocabulary words.
  2. The authors propose encoding rare and unknown words as sequences of subword units.
  3. The approach builds on the intuition that names, compounds, loanwords, and cognates can be translated using sub-word morphological or character components.
  4. Evaluated segmentation methods include simple character n-gram models and Byte Pair Encoding (BPE).
  5. On WMT 15 benchmarks, subword models outperformed dictionary back-off baselines by 1.1 BLEU on English-German and 1.3 BLEU on English-Russian.

Why it matters

Subword tokenization via Byte Pair Encoding addresses the out-of-vocabulary limitation directly, serving as a core building block for modern text representation in NLP and contemporary language models.

Analysis

The paper demonstrates that translation models do not need to adhere strictly to conventional word boundaries to retain morphological meaning. By applying compression techniques like BPE to vocabulary segmentation, the authors achieved consistent empirical improvements without relying on secondary lookup dictionaries.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

    Neural Machine Translation by Jointly Learning to Align and Translate

    In this paper, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio propose an approach that resolves the fixed-length vector bottleneck in encoder-decoder translation models through soft search.

    AI & LLMs· Research

  • Article:Wikipedia contributors

    Byte-Pair Encoding: From Data Compression to LLM Tokenization

    Byte-pair encoding (BPE), originally introduced in 1994 by Philip Gage for data compression, serves in a modified form as the foundation for modern large language model tokenizers. Rather than aiming for maximum compression, the LLM adaptation builds a fixed token vocabulary by iteratively merging frequent n-grams.

    AI & LLMs· Resource

  • Article:Yonghui Wu, Mike Schuster, Zhifeng Chen, et al.

    Google Introduces GNMT: Neural Machine Translation Built for Production

    Google researchers present GNMT (Google's Neural Machine Translation system), addressing long-standing obstacles in neural translation such as high compute costs during training, slow inference, and difficulties handling rare words.

    AI & LLMs· Research

  • Article:Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

    Efficient Estimation of Word Representations in Vector Space

    A research team led by Tomas Mikolov introduces two novel model architectures designed to compute continuous vector representations of words from massive datasets. The approach delivers higher accuracy with substantially lower computational requirements than prior neural network methods.

    AI & LLMs· Research

  • Article:Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

    Attention Is All You Need: Introducing the Transformer Architecture

    The authors introduce the Transformer, a network architecture that completely replaces recurrence and convolutions with attention mechanisms.

    AI & LLMs· Research

  • Article:Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean

    Distributed Representations of Words and Phrases and their Compositionality

    The authors present extensions to the continuous Skip-gram model to improve vector quality and training speed. The work introduces subsampling of frequent words, negative sampling as an alternative to hierarchical softmax, and a technique for learning distributed representations of idiomatic phrases.

    AI & LLMs· Research

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.