Mamba: Linear-Time Sequence Modeling with Selective State Spaces

ArticleAlbert Gu, Tri DaoResearch

Albert Gu and Tri Dao introduce Mamba, a sequence modeling architecture based on selective structured state space models (SSMs). It achieves linear scaling with sequence length, eliminates attention and MLP blocks, and delivers five times higher inference throughput than Transformers.

Key points

  1. Prior subquadratic models struggled on language tasks due to an inability to perform content-based reasoning.
  2. Mamba makes SSM parameters input-dependent, enabling selective propagation or forgetting of information along the sequence.
  3. A hardware-aware parallel algorithm operates in recurrent mode to maintain high efficiency without standard convolutions.
  4. The simplified end-to-end architecture functions entirely without attention modules or MLP blocks.
  5. Mamba-3B outperforms equal-sized Transformers and matches the performance of models twice its size across pretraining and downstream benchmarks.
  6. The architecture achieves state-of-the-art results across language, audio, and genomics, scaling effectively to million-length sequences.

Why it matters

Standard Transformers face quadratic computational bottlenecks on long inputs. Mamba provides an efficient, linearly scaling alternative that maintains high output quality and dramatically increases inference throughput.

Analysis

By addressing the fundamental limitation of previous state space models—the inability to dynamically filter and focus on relevant context—Mamba bridges the gap between linear-time architectures and attention. Combined with custom hardware execution strategies, it establishes SSMs as a credible foundation model backbone.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Opher Lieber, Barak Lenz, Hofit Bata et al.

    Jamba: A Hybrid Transformer-Mamba Language Model

    Researchers introduce Jamba, a base large language model utilizing a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. By interleaving Transformer and Mamba layers with MoE modules, Jamba fits into a single 80GB GPU while supporting context lengths of up to 256K tokens.

    AI & LLMs· Research

  • Article:Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter

    xLSTM: Extended Long Short-Term Memory

    A research team led by Sepp Hochreiter introduces xLSTM, an extension of the original Long Short-Term Memory architecture designed for large-scale language modelling. The approach addresses historical scaling limits of LSTMs and demonstrates competitive performance against state-of-the-art Transformers and State Space Models.

    AI & LLMs· Research

  • Article:Kimi Team (Yu Zhang et al.)

    Kimi Linear: An Expressive Hybrid Linear Attention Architecture

    The Kimi Team presents Kimi Linear, a hybrid linear attention architecture that outperforms standard full attention across short-context, long-context, and RL scaling scenarios. It cuts KV cache consumption by up to 75% and boosts decoding throughput up to sixfold on 1M token contexts.

    AI & LLMs· Research

  • Link:Subquadratic Inc.

    Subquadratic Unveils SubQ: LLM Architecture for Multi-Million Token Contexts

    AI research company Subquadratic has introduced SubQ, a model designed for multi-million token reasoning. By using a proprietary algorithm called Subquadratic Sparse Attention (SSA), the company aims to significantly cut compute and memory demands compared to traditional Transformer-based architectures on large contexts.

    AI & LLMs· Announcement

  • Article:Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

    Attention Is All You Need: Introducing the Transformer Architecture

    The authors introduce the Transformer, a network architecture that completely replaces recurrence and convolutions with attention mechanisms.

    AI & LLMs· Research

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.