Distributed Representations of Words and Phrases and their Compositionality

ArticleTomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey DeanResearch

The authors present extensions to the continuous Skip-gram model to improve vector quality and training speed. The work introduces subsampling of frequent words, negative sampling as an alternative to hierarchical softmax, and a technique for learning distributed representations of idiomatic phrases.

Key points

  1. Subsampling frequent words delivers notable speedups and results in more regular word representations.
  2. Negative sampling is introduced as a straightforward alternative to hierarchical softmax.
  3. A phrase-finding method addresses the limitation of word models regarding idiomatic expressions such as 'Air Canada'.
  4. The model enables the learning of vector representations across millions of phrases.

Why it matters

The optimizations established efficient mechanisms for scaling vector representations, addressing computational bottlenecks and the compositional limits of word-level semantics.

Analysis

Mikolov, Sutskever and their co-authors identify core constraints in word-based vector spaces, specifically compute costs and the failure of compositionality in idioms. By introducing negative sampling and phrase detection, the paper details practical steps to train high-quality embeddings at massive scale.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

    Efficient Estimation of Word Representations in Vector Space

    A research team led by Tomas Mikolov introduces two novel model architectures designed to compute continuous vector representations of words from massive datasets. The approach delivers higher accuracy with substantially lower computational requirements than prior neural network methods.

    AI & LLMs· Research

  • Article:Rico Sennrich, Barry Haddow, Alexandra Birch

    Neural Machine Translation of Rare Words with Subword Units

    Rico Sennrich, Barry Haddow, and Alexandra Birch present an approach enabling neural machine translation (NMT) systems to handle rare and out-of-vocabulary words using subword units. Instead of fixed vocabularies and dictionary back-offs, the technique employs segmentation algorithms like Byte Pair Encoding (BPE) for open-vocabulary translation.

    AI & LLMs· Research

  • Article:Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

    Neural Machine Translation by Jointly Learning to Align and Translate

    In this paper, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio propose an approach that resolves the fixed-length vector bottleneck in encoder-decoder translation models through soft search.

    AI & LLMs· Research

  • Link:Xavier Glorot, Yoshua Bengio

    Understanding the Difficulty of Training Deep Feedforward Neural Networks

    This research paper by Xavier Glorot and Yoshua Bengio investigates why standard gradient descent with random initialization performs poorly in deep neural networks. The authors examine the role of activation functions and gradient behaviour across layers, proposing a new initialization scheme that significantly accelerates convergence.

    AI & LLMs· Research

  • Article:Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter

    xLSTM: Extended Long Short-Term Memory

    A research team led by Sepp Hochreiter introduces xLSTM, an extension of the original Long Short-Term Memory architecture designed for large-scale language modelling. The approach addresses historical scaling limits of LSTMs and demonstrates competitive performance against state-of-the-art Transformers and State Space Models.

    AI & LLMs· Research

  • Article:Wikipedia contributors

    John Rupert Firth: The Linguist Behind the Foundation of Modern Word Embeddings

    British linguist J. R. Firth (1890–1960) shaped modern natural language processing with his famous maxim: 'You shall know a word by the company it keeps'. His work on distributional semantics provided the theoretical cornerstone for vector representations like Word2vec and GloVe.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.