Serving DeepSeek V4 Flash on Dual DGX Spark Nodes

ArticleElite Test EngineeringTutorial

A detailed technical setup report documents serving DeepSeek-V4-Flash (284B MoE) across two DGX Spark nodes powered by Grace Blackwell chips using Ray and vLLM. It highlights workarounds for unified memory monitoring, NCCL RDMA deadlocks, and CUDA graph issues, as well as a 45% speedup using native speculative decoding.

Key points

  1. The NVFP4 checkpoint hit an unfixable SM121 kernel selection gap in vLLM, prompting a switch to the official FP8 checkpoint (159.6 GB) at TP=2.
  2. Cross-node NCCL hangs over QSFP/ConnectX-7 were resolved via NCCL_IB_DISABLE=1 (falling back to TCP), and Ray's host memory monitor was disabled for unified memory.
  3. CUDA graph captures corrupted cross-node token replays during decoding, making --enforce-eager mandatory for correct outputs.
  4. The fp8 KV cache dtype is strictly required by the model's MLA layout; non-default block sizes failed due to sliding-window constraints.
  5. Enabling native Multi-Token Prediction (MTP) draft heads with speculative depth 2 raised single-stream throughput from ~17–18 tok/s to ~25.3 tok/s.

Why it matters

Deploying cutting-edge 284B MoE models on compact DGX clusters reveals friction points between bleeding-edge Grace Blackwell (SM121) hardware, networking libraries, and inference engines. This setup report provides verified parameter configurations and workarounds.

Analysis

The report illustrates the fragility of bleeding-edge inference stacks: Ray's OOM monitors clash with unified host-GPU memory, and NCCL's IB-verbs transport deadlocks over ConnectX-7 fabrics. Sacrificing CUDA graphs via --enforce-eager is an unfortunate hit to raw compute efficiency, but necessary to prevent silent token corruption. Crucially, the presence of native MTP draft heads within DeepSeek V4 allows speculative decoding without extra memory overhead, offsetting the eager-mode penalty with a 45% net throughput gain.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Video

    Video:EliteTestEngineering

    Testing DeepSeek V4 Flash on a Clustered DGX Spark and Acer GN100

    EliteTestEngineering demonstrates running DeepSeek V4 Flash across a two-node cluster consisting of a DGX Spark and an Acer GN100. The MoE model achieves inference speeds of 10 to 16.5 tokens per second but quickly encounters memory limitations with longer context windows.

    AI & LLMs· Demo

  • Repository:MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    Running DeepSeek-v4-Flash on a Single DGX Spark

    An open-source launcher and Docker recipe for running DeepSeek V4 Flash 0731 (EXL3) on a single NVIDIA DGX Spark with 128 GiB unified memory.

    328starsPython

    AI & LLMs· Tool

  • X post:Mia

    Running DeepSeek v4 Flash on a Single NVIDIA DGX Spark

    MiaAI-Lab published an open-source launcher allowing DeepSeek V4 Flash 0731 (EXL3) to run on a single NVIDIA DGX Spark node instead of requiring a multi-node setup.

    71bookmarks9.8kviews

    AI & LLMs· Tool

  • Repository:MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    Serving GLM-5.3 Flash EXL3 on Dual NVIDIA DGX Spark Kits

    Mia's AI Lab provides an optimised vLLM deployment overlay for GLM-5.3-Flash using 4 bpw EXL3 quantisation tailored for 2x NVIDIA GB10 systems.

    431starsPython

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.