ArticleKimi Team (Yu Zhang et al.)Research
Full attention mechanisms scale quadratically and create memory bottlenecks at long context lengths. Kimi Linear demonstrates that linear attention can serve as a viable drop-in replacement without compromising model quality across tasks.
Linear attention models traditionally struggled to match standard attention accuracy across varied tasks. By combining improved gating in KDA, efficient chunkwise DPLR computation, and hybridization with MLA, the researchers provide a scalable architecture ready for production inference via vLLM.
Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
X post:AYi
A widely discussed thread breaks down the architectural progression from GPT-2 to Moonshot AI's open-source Kimi K3, illustrating how linear attention and DeltaNet resolve transformer memory bottlenecks beyond simple scaling.
1.9kbookmarks402kviews
Article:Albert Gu, Tri Dao
Albert Gu and Tri Dao introduce Mamba, a sequence modeling architecture based on selective structured state space models (SSMs). It achieves linear scaling with sequence length, eliminates attention and MLP blocks, and delivers five times higher inference throughput than Transformers.
Link:Moonshot AI
Moonshot AI has released Kimi K3, an open-weight native multimodal agentic model featuring 2.8 trillion total parameters and a 1-million-token context window. The architecture uses a sparse MoE setup that activates 104 billion parameters per token.
Link:Subquadratic Inc.
AI research company Subquadratic has introduced SubQ, a model designed for multi-million token reasoning. By using a proprietary algorithm called Subquadratic Sparse Attention (SSA), the company aims to significantly cut compute and memory demands compared to traditional Transformer-based architectures on large contexts.
Article:Opher Lieber, Barak Lenz, Hofit Bata et al.
Researchers introduce Jamba, a base large language model utilizing a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. By interleaving Transformer and Mamba layers with MoE modules, Jamba fits into a single 80GB GPU while supporting context lengths of up to 256K tokens.
Link:inclusionAI
inclusionAI has introduced Ling-3.0-tiny, a compact MoE model with 7.9B total parameters activating only 1.3B parameters per token. Utilizing a hybrid architecture of linear and latent attention mechanisms, it aims to deliver efficient reasoning and agentic workloads on local hardware.
Parts of this content were created with the assistance of AI.
