ArticleOpher Lieber, Barak Lenz, Hofit Bata et al.Research
Pure Transformer architectures face significant memory and compute bottlenecks as context windows expand. Jamba demonstrates that blending Mamba state-space models with attention layers and MoE can substantially lower inference resource requirements while maintaining competitive benchmark performance.
The authors explore architectural trade-offs in combining Transformer and Mamba components, highlighting design choices necessary for stable large-scale training. By showing that 256K token contexts can be handled within an 80GB hardware budget, Jamba provides practical evidence for the viability of state-space hybrid models. Making weights and ablation checkpoints openly available aids further community experimentation.
Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Article:Albert Gu, Tri Dao
Albert Gu and Tri Dao introduce Mamba, a sequence modeling architecture based on selective structured state space models (SSMs). It achieves linear scaling with sequence length, eliminates attention and MLP blocks, and delivers five times higher inference throughput than Transformers.
Article:Kimi Team (Yu Zhang et al.)
The Kimi Team presents Kimi Linear, a hybrid linear attention architecture that outperforms standard full attention across short-context, long-context, and RL scaling scenarios. It cuts KV cache consumption by up to 75% and boosts decoding throughput up to sixfold on 1M token contexts.
Repository:inclusionAI/Ling
InclusionAI has open-sourced Ling, a family of Mixture-of-Experts (MoE) large language models. The suite is offered in two sizes: Ling-lite with 16.8 billion total parameters (2.75 billion activated) and Ling-plus with 290 billion total parameters (28.8 billion activated).
267starsPython
Article:Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter
A research team led by Sepp Hochreiter introduces xLSTM, an extension of the original Long Short-Term Memory architecture designed for large-scale language modelling. The approach addresses historical scaling limits of LSTMs and demonstrates competitive performance against state-of-the-art Transformers and State Space Models.
Article:Nikita Pavlichenko
JetBrains has released Mellum2, a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. Activating only 2.5B parameters per token and published under the Apache 2.0 license, it is designed for low-latency, high-throughput developer and AI workloads.
Link:drumih
TurboFieldfare is an open-source inference engine written in Swift and Metal that executes 4-bit Gemma 4 26B-A4B-IT on Apple M-series Macs using roughly 2 GB of RAM. While the model weights require 14 GB of disk space, the engine keeps only the shared model base and KV cache in memory and streams routed expert layers on demand directly from the SSD.
Parts of this content were created with the assistance of AI.
