DeepSeek V4-Pro Architecture: KV Cache Compression and Hyper-Connections Visualised

ArticleSamuel BatistaResource

Samuel Batista breaks down the inner mechanics of DeepSeek V4-Pro across 61 layers and a 1-million-token context window. The visual breakdown explains compressed attention, four-lane hyper-connections, and sparse MoE routing. It shows how the model cuts KV cache footprint while scaling parameters.

Key points

  1. DeepSeek V4-Pro handles a 1-million-token context using only 10% of the KV cache and 27% of compute per generated token compared to V3.2.
  2. Heavily Compressed Attention (HCA) aggregates 128 tokens per entry, while Compressed Sparse Attention (CSA) compresses groups of four.
  3. A lightning indexer filters roughly 262,000 compressed entries down to the top 1,024 for CSA layers, backed by a 128-token uncompressed sliding window.
  4. Manifold-constrained hyper-connections (mHC) route tokens across four parallel residual streams using stabilized 4x4 mixing grids.
  5. From 384 routed MoE experts, only six run alongside one shared expert per layer, activating ~49 billion out of 1.6 trillion parameters per token.

Why it matters

Understanding DeepSeek V4's architecture highlights practical engineering solutions for long-context LLMs, trading exact attention for tiered compression to curb memory costs.

Analysis

The Million-Token Memory Problem

Full attention across a million tokens typically demands massive KV cache storage—hundreds of gigabytes per conversation. DeepSeek V4-Pro addresses this challenge through tiered cache compression and sparse retrieval.

Dual Compression and Indexing

To keep context manageable, V4 splits token history into multiple representations:

  • Heavily Compressed Attention (HCA): Merges 128 tokens into single entries using learned weighted averages.
  • Compressed Sparse Attention (CSA): Retains finer granularity by bundling four tokens at a time.
  • Lightning Indexer: Performs a low-precision, cheap similarity pass over ~262,000 compressed entries to pick the top 1,024 most relevant candidates for CSA layers.
  • Sliding Window: Preserves uncompressed key-value pairs for the most recent 128 tokens, ensuring zero detail loss for immediate dialogue.

Hyper-Connections and MoE Scaling

Rather than passing a single residual stream, V4 routes representations through four parallel numerical lists. DeepSeek balances this via manifold-constrained hyper-connections (mHC), using constrained 4x4 matrix weights whose rows and columns sum to approximately one, preventing runaway amplification across layers.

For inference compute efficiency, V4 utilizes 384 routed experts per layer alongside one shared expert. Each token activates only six routed experts, keeping the active parameter count at ~49 billion out of 1.6 trillion. Unused experts are stored in 4-bit precision.

Retrieval Limitations

The primary compromise involves detail retention. According to DeepSeek's MRCR retrieval tests, retrieval remains reliable up to approximately 128,000 tokens before performance starts degrading, reflecting the lossy nature of both HCA compression and indexer filtering.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 10 November 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:DeepSeek AI

    DeepSeek Launches DeepSeek-V4 Preview as Open Source with 1M Context

    DeepSeek has open-sourced the preview release of DeepSeek-V4. The line-up features DeepSeek-V4-Pro and DeepSeek-V4-Flash, standardising a 1M context window and introducing architectural innovations to cut compute and memory overhead.

    AI & LLMs· Announcement

  • Link:deepseek-ai

    NVIDIA Build lists DeepSeek-V4-Flash-0731 MoE model

    DeepSeek-V4-Flash-0731 is a sparse Mixture-of-Experts language model featuring 13B active parameters and a 1-million-token context window. The NVIDIA Build model card outlines technical specifications, architecture details, and coding benchmark performances.

    AI & LLMs· Resource

  • Repository:Tencent-Hunyuan/Hy4-preview

    Tencent Hunyuan Unveils Hy4-preview: 770B MoE Model with Sparse Attention

    The Tencent Hy Team has introduced Hy4-preview, a new flagship Mixture-of-Experts (MoE) model featuring 770B total parameters with 49B activated per token. The architecture incorporates MoE routing, Gated DeepSeek Sparse Attention, and a native MTP layer for speculative decoding.

    344starsPython

    AI & LLMs· Announcement

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.