vLLM on NVIDIA DGX Spark: Architecture, Configuration, and Local Evaluation

ArticleInferactTutorial

This guide details running vLLM on NVIDIA's DGX Spark (GB10) desk-side workstation. Leveraging its 128 GB unified memory pool, the system is designed for local inference with NVFP4 MoE models such as Nemotron-3-Super.

Key points

  1. DGX Spark features the GB10 SoC (sm_121 Blackwell architecture) with a shared 128 GB unified memory pool for CPU, GPU, OS, and containers.
  2. The hardware profile fits NVFP4 Mixture-of-Experts models with roughly 10–15B active parameters, such as Nemotron-3-Super-120B-A12B-NVFP4.
  3. Runtime settings like --gpu-memory-utilization must leave sufficient headroom in unified memory for the OS, runtime, and KV cache expansion.
  4. The system targets single-user and low-concurrency local inference rather than high-throughput data-centre workloads, requiring a low --max-num-seqs setting.
  5. vLLM exposes an OpenAI-compatible API; token streaming and paged KV cache ensure low perceived latency for interactive workflows.
  6. Deployments rely on official sm_121-compatible images (e.g. vllm/vllm-openai:cu130-nightly) and pre-staged model weights via host mounts.

Why it matters

DGX Spark bridges the gap between laptop-scale prototyping and server-grade GPU clusters. Combined with vLLM and NVFP4, it allows teams to run models exceeding 100 billion parameters locally on desk-side hardware without third-party cloud dependencies.

Analysis

The architectural focal point of DGX Spark is its unified memory pool: because CPU and GPU share the 128 GB memory, standard server-oriented vLLM configurations can exhaust memory rapidly. The post demonstrates that continuous batching, paged KV cache, and disciplined flag tuning make local MoE inference feasible, provided workloads remain oriented around low-concurrency interactive tasks rather than massive parallel serving.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Video

    Video:Heavy Metal Cloud

    Nvidia DGX Spark Hands-On: Setup, vLLM Serving, and Mac Studio Comparison

    Heavy Metal Cloud reviews the Gigabyte and ASUS Nvidia DGX Spark desktop units powered by the Grace Blackwell architecture. The video provides a step-by-step guide for local LLM serving using Docker, vLLM, and SparkRun, as well as media generation in ComfyUI, comparing real-world performance against a Mac Studio.

    AI & LLMs· Tutorial

  • Repository:eugr/spark-vllm-docker

    spark-vllm-docker: Optimised vLLM Docker Setups for DGX Spark

    spark-vllm-docker is an open-source tool providing Docker configurations and startup scripts to run vLLM on DGX Spark systems, supporting both single-node setups and multi-node clusters via Ray or PyTorch distributed mode.

    2.2kstarsShell

    AI & LLMs· Tool

  • Article:Elite Test Engineering

    Serving DeepSeek V4 Flash on Dual DGX Spark Nodes

    A detailed technical setup report documents serving DeepSeek-V4-Flash (284B MoE) across two DGX Spark nodes powered by Grace Blackwell chips using Ray and vLLM. It highlights workarounds for unified memory monitoring, NCCL RDMA deadlocks, and CUDA graph issues, as well as a 45% speedup using native speculative decoding.

    AI & LLMs· Tutorial

  • Repository:MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    Running DeepSeek-v4-Flash on a Single DGX Spark

    An open-source launcher and Docker recipe for running DeepSeek V4 Flash 0731 (EXL3) on a single NVIDIA DGX Spark with 128 GiB unified memory.

    328starsPython

    AI & LLMs· Tool

  • Video

    Video:AIProgrammingHardware

    Essential Software Ecosystem and GitHub Repositories for NVIDIA DGX Spark

    This video details the primary open-source software repositories and workflows for NVIDIA's DGX Spark desktop AI supercomputer. Powered by the GB10 Grace Blackwell Superchip with 128 GB of unified memory, the hardware enables private local LLM inference, multi-agent setups, and model training without cloud reliance.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.