Hybrid LLM Inference Setup: DGX Sparks for Prefill and M3 Ultra for Decode

X postXaden RyanDiscussion

Xaden Ryan is experimenting with a split LLM pipeline: two DGX Sparks will handle the compute-heavy prefill phase, followed by transferring the KV cache to a 512 GB Apple M3 Ultra for decoding, targeting 200k-token contexts in GLM-5.3.

Key points

  1. Ryan plans to perform large prefills across two DGX Sparks and transfer the computed KV cache over USB to an M3 Ultra (512 GB) for decoding.
  2. While the original post mentioned Deepseek-V4-Flash, Ryan clarified in the thread that he is targeting GLM-5.3 for superior reasoning capabilities.
  3. The setup relies on a USB connection approach developed by ashxhart and a kernel patch contributed by exolabs.
  4. The motivation comes from severe prefill slowdowns on the M3 Ultra alone, where processing a 200k context prompt reportedly takes hours.
  5. Ryan is testing the workflow on an M3 Ultra despite documentation stating support is limited to M4 or newer hardware.

Why it matters

Long-context inference faces severe compute bottlenecks during the prompt-processing phase on unified memory architectures. Distributing the pipeline between fast accelerators for prefill and high-capacity unified memory for decode could make local execution of large models practical.

Analysis

The experiment highlights current hardware trade-offs in local AI inference. While Apple Silicon provides the memory capacity required for large context windows, its prefill throughput lags behind dedicated accelerators. Ryan acknowledges that it remains uncertain whether bandwidth and compatibility will hold up, noting that future hardware like an M5 Ultra might eventually eliminate the need for such hybrid workarounds.

Original post

Xaden Ryan

@XadenRyan · 5 September 2026

2x DGX Sparks secured. Goal is large prefill done on 2x DGX Sparks and then transferring over to M3 Ultra 512 GB to do decode for Deepseek-V4-Flash. Going to try getting this working over the usb connect @ashxhart has been working on and the kernel patch @exolabs got in. t.co/FTftBb6tLI t.co/UX4mrPyazu

84 likes21 replies25 bookmarks8.8k views

View on X
More posts in the thread (9)
  1. @0xkydo @ashxhart @exolabs On the website it says it only supports M4 or newer but I’m gonna test it anyway
  2. @GumbiiDigital @ashxhart @exolabs Sorry I was looking forward to meeting up but I was able to snag some tickets last minute to a van life festival in Oregon that I’ve been wanting to go to for years. Now I’m booking it 3000 miles across the country for this thing next week. I might be back in December tho
  3. @seanhighness @ashxhart @exolabs Maybe, I’m going to try some different experiments. Right now full GLM-5.3 prefill is to slow with a 200k context on the m3 ultra alone (it’s like hours for a single 200k token prompt). So I’m hoping to speed that up with the DGX Sparks. I have no idea if it’s actually gonna work
  4. @en4ble1337 @ashxhart @exolabs Yeah have to send computed kv cache over
  5. @ItsCuthulhu @ashxhart @exolabs I’ve been running it in my 128 GB M4 and the code is definitely cooking. I want something that does reasoning better. So I’m thinking GLM-5.3
  6. @ItsCuthulhu I’m also realizing now that I put DeepSeek on the original post and not GLM-5.3. Whoops.
  7. @ItsCuthulhu Sweet! Thanks!
  8. @spactastical No, It shouldn’t be
  9. @CMichaelGibbs I suspect with the M5 Ultra you won’t need to do this as as it can do much faster prefill
Selected replies (5)
  • @ItsCuthulhu @XadenRyan @ashxhart @exolabs t.co/Kndfupi6zN You could run 3 or more Qwen 3.8 Flash Next on all that. Have them always on. At this point, it's just a better model. Love Deepseek but this is the better model, smaller, cheaper, and you can have many of them running on Hermes. t.co/dWKFXIfb4L
  • @0xkydo @XadenRyan @ashxhart @exolabs does nvidia's pair work here?
  • @GumbiiDigital @XadenRyan @ashxhart @exolabs Sweet! I’ve got extra cables if you need them…when you headed down my way?
  • @ItsCuthulhu @XadenRyan t.co/F9QU2VPtMJ This may be the best bet for now!
  • @CMichaelGibbs @XadenRyan @ashxhart @exolabs Really keen to see how this works. My M5 Ultra should be here in about 4 weeks with 256. I have 4 sparks and am really keen to see if we can have the best of both worlds :D Keep in touch and let me know if you need/want another set of eyes.

Mentioned links

Links and tools from this entry

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Repository:MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    Serving GLM-5.3 Flash EXL3 on Dual NVIDIA DGX Spark Kits

    Mia's AI Lab provides an optimised vLLM deployment overlay for GLM-5.3-Flash using 4 bpw EXL3 quantisation tailored for 2x NVIDIA GB10 systems.

    431starsPython

    AI & LLMs· Tool

  • Repository:MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    Running DeepSeek-v4-Flash on a Single DGX Spark

    An open-source launcher and Docker recipe for running DeepSeek V4 Flash 0731 (EXL3) on a single NVIDIA DGX Spark with 128 GiB unified memory.

    328starsPython

    AI & LLMs· Tool

  • X post:Mia

    Running DeepSeek v4 Flash on a Single NVIDIA DGX Spark

    MiaAI-Lab published an open-source launcher allowing DeepSeek V4 Flash 0731 (EXL3) to run on a single NVIDIA DGX Spark node instead of requiring a multi-node setup.

    71bookmarks9.8kviews

    AI & LLMs· Tool

  • Video

    Video:Heavy Metal Cloud

    Nvidia DGX Spark Hands-On: Setup, vLLM Serving, and Mac Studio Comparison

    Heavy Metal Cloud reviews the Gigabyte and ASUS Nvidia DGX Spark desktop units powered by the Grace Blackwell architecture. The video provides a step-by-step guide for local LLM serving using Docker, vLLM, and SparkRun, as well as media generation in ComfyUI, comparing real-world performance against a Mac Studio.

    AI & LLMs· Tutorial

  • X post:Tech2Wild

    Optimizing Dual GPU Setups for Large Language Models

    This post explores a technical setup for running large language models using dual GPU configurations, specifically highlighting the use of hardware like the Spark and dual 4090s. The author provides a detailed guide on managing context windows and model instances to improve performance and efficiency for local AI workflows.

    52bookmarks8.7kviews

    AI & LLMs

  • Article:Elite Test Engineering

    Serving DeepSeek V4 Flash on Dual DGX Spark Nodes

    A detailed technical setup report documents serving DeepSeek-V4-Flash (284B MoE) across two DGX Spark nodes powered by Grace Blackwell chips using Ray and vLLM. It highlights workarounds for unified memory monitoring, NCCL RDMA deadlocks, and CUDA graph issues, as well as a 45% speedup using native speculative decoding.

    AI & LLMs· Tutorial

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.