Article:Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean
Distributed Representations of Words and Phrases and their Compositionality
The authors present extensions to the continuous Skip-gram model to improve vector quality and training speed. The work introduces subsampling of frequent words, negative sampling as an alternative to hierarchical softmax, and a technique for learning distributed representations of idiomatic phrases.
From Video: Building an LLM from Scratch: Neural Networks, Tokenization, and Transformers