4 entries
Keyword: transformerArticle:Opher Lieber, Barak Lenz, Hofit Bata et al.
Researchers introduce Jamba, a base large language model utilizing a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. By interleaving Transformer and Mamba layers with MoE modules, Jamba fits into a single 80GB GPU while supporting context lengths of up to 256K tokens.
From Video: Building an LLM from Scratch: Neural Networks, Tokenization, and Transformers
Article:Wikipedia contributors
This Wikipedia overview covers the mechanics and historical evolution of transformer models. Built upon multi-head attention, the architecture abandons recurrence, enabling highly parallel training on GPUs.
From Video: Building an LLM from Scratch: Neural Networks, Tokenization, and Transformers
Article:Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The authors introduce the Transformer, a network architecture that completely replaces recurrence and convolutions with attention mechanisms.
From Video: Building an LLM from Scratch: Neural Networks, Tokenization, and Transformers
VideoVideo:CURT
CJ breaks down the inner workings of large language models using historical context and working code implementations. He guides through the evolution from rule-based chatbots to transformers, covering neural networks, Byte Pair Encoding, and vector embeddings.
4 of 4 shown
Do you have an interesting article, a video, or a tweet that belongs in our knowledge base? Write to us at office@webconsulting.at.
Parts of this content were created with the assistance of AI.
