ArticleMaximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp HochreiterResearch
While Transformer architectures currently underpin most LLMs due to efficient training parallelization, xLSTM demonstrates that recurrent architectures can overcome historic bottlenecks and serve as a viable alternative at scale.
By modifying memory structures and update mechanisms—specifically through matrix memory in mLSTM and exponential gating—the researchers systematically remove the sequential execution bottlenecks of traditional LSTMs. The resulting architectures maintain recurrent advantages while scaling competitively alongside dominant attention-based systems.
Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Article:Albert Gu, Tri Dao
Albert Gu and Tri Dao introduce Mamba, a sequence modeling architecture based on selective structured state space models (SSMs). It achieves linear scaling with sequence length, eliminates attention and MLP blocks, and delivers five times higher inference throughput than Transformers.
Article:Wikipedia contributors
This Wikipedia overview covers the mechanics and historical evolution of transformer models. Built upon multi-head attention, the architecture abandons recurrence, enabling highly parallel training on GPUs.
Article:Opher Lieber, Barak Lenz, Hofit Bata et al.
Researchers introduce Jamba, a base large language model utilizing a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. By interleaving Transformer and Mamba layers with MoE modules, Jamba fits into a single 80GB GPU while supporting context lengths of up to 256K tokens.
Repository:test-time-training/e2e
This repository provides the official JAX implementation of End-to-End Test-Time Training (TTT) for long contexts. It approaches long-context language modeling as a continual learning challenge, using a standard Transformer with sliding-window attention that compresses input contexts into its weights at test time via next-token prediction.
693starsPython
Article:Kimi Team (Yu Zhang et al.)
The Kimi Team presents Kimi Linear, a hybrid linear attention architecture that outperforms standard full attention across short-context, long-context, and RL scaling scenarios. It cuts KV cache consumption by up to 75% and boosts decoding throughput up to sixfold on 1M token contexts.
Article:Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The authors introduce the Transformer, a network architecture that completely replaces recurrence and convolutions with attention mechanisms.
Parts of this content were created with the assistance of AI.
