LLM Inference in Production: Silent GPU OOMs and Health Check Blind Spots

X postNeko LegendsDiscussion

Neko Legends shares findings on running GLM-5.3-Flash (1M-ctx) on a 4× Spark cluster. The cluster failed after 13 hours due to a silent GPU OOM lasting 30 minutes, despite the /health endpoint reporting normal operation throughout.

Key points

  1. A 4× Spark cluster running GLM-5.3-Flash (1M context) crashed after 13 hours due to a silent GPU OOM that went undetected for 30 minutes.
  2. The /health endpoint reported green throughout the outage because the process had not exited.
  3. GPU memory allocation was trimmed from 85% with zero margin down to 80% to provide operational buffer.
  4. Request logging is enabled but capped to prevent disk exhaustion.
  5. Crash evidence is preserved before cleanup tasks run, accompanied by kernel-level memory error monitoring.

Why it matters

Inference benchmarks routinely highlight tokens per second while ignoring long-term stability and uptime. Without hardware-aware health probes, silent VRAM failures can disrupt live services without triggering alerts.

Analysis

The scenario exposes a key flaw in standard process-based health probes for LLM serving: a hung or OOM-locked GPU worker does not necessarily crash the host process. Production reliability requires VRAM headroom, explicit GPU state monitoring, and preservation of crash traces.

Original post

Neko Legends

@softpoo · 4 September 2026

• Gave the GPUs breathing room — they were running at 85% memory with zero slack, now 80% • Logs every request, capped so it can't fill the disk • Saves crash evidence before cleanup deletes it (we lost the first corpse) • Watches each machine's kernel for memory errors —

6 likes1 replies18 bookmarks683 views

View on X
More posts in the thread (2)
  1. Everyone posts tok/s. Nobody posts uptime. Our 4× Spark cluster runs GLM-5.3-Flash 1M-ctx as a daily driver — real chats, not benches. It died after 13 hours on a "timeout" that was really silent GPU OOM for 30 min. /health said fine the whole time. Grab update fix below t.co/QJXquOkUGB
  2. @DeltaTrad3r its not what i was really going for - not the number - but stability. crashing is unacceptable regardless of how long it is up.
Selected replies (5)
  • @DeltaTrad3r @softpoo I was going for uptime as a metric in my systems but sometimes I want to install firmware updates and that requires a reboot
  • @BTCXoomer @softpoo Nice work. I can’t wait to install it!
  • @doth4580 @softpoo 100% agreed. Influencers who give outbthese recipes should at least do a 24hour endurance test
  • @AIQuanting @softpoo health said fine because it wasnt looking at vram. an oom that doesnt crash the process will always pass a process-alive check, so the 13 hours of uptime had a 30 minute hole in it that the dashboard couldnt see
  • @techiedheeraj @softpoo steps to setup ?

Mentioned links

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Elite Test Engineering

    Serving DeepSeek V4 Flash on Dual DGX Spark Nodes

    A detailed technical setup report documents serving DeepSeek-V4-Flash (284B MoE) across two DGX Spark nodes powered by Grace Blackwell chips using Ray and vLLM. It highlights workarounds for unified memory monitoring, NCCL RDMA deadlocks, and CUDA graph issues, as well as a 45% speedup using native speculative decoding.

    AI & LLMs· Tutorial

  • X post:Wei

    Dual DGX Spark Clock Sweep: Finding Efficiency at 1800 MHz

    Wei benchmarked a dual DGX Spark setup across clock speeds from 2200 MHz down to 1400 MHz to identify an optimal production profile. Setting the clock to 1800 MHz reduced average power consumption by approximately 30% and peak temperatures by 5–7 °C while sacrificing only minimal performance.

    163bookmarks7.8kviews

    AI & LLMs· Research

  • Repository:tonyd2wild/The-Sparky-Command-Center

    Sparky Command Center: Dependency-Free Monitoring for GPU Fleets and LLM Inference

    Sparky Command Center is a lightweight, zero-dependency web dashboard designed to monitor DGX Spark fleets and custom Linux GPU rigs. It collects hardware metrics over SSH and scrapes live inference performance data from vLLM and llama.cpp without external databases.

    40starsPython

    AI & LLMs· Tool

  • Video

    Video:Heavy Metal Cloud

    Nvidia DGX Spark Hands-On: Setup, vLLM Serving, and Mac Studio Comparison

    Heavy Metal Cloud reviews the Gigabyte and ASUS Nvidia DGX Spark desktop units powered by the Grace Blackwell architecture. The video provides a step-by-step guide for local LLM serving using Docker, vLLM, and SparkRun, as well as media generation in ComfyUI, comparing real-world performance against a Mac Studio.

    AI & LLMs· Tutorial

  • X post:Xaden Ryan

    Hybrid LLM Inference Setup: DGX Sparks for Prefill and M3 Ultra for Decode

    Xaden Ryan is experimenting with a split LLM pipeline: two DGX Sparks will handle the compute-heavy prefill phase, followed by transferring the KV cache to a 512 GB Apple M3 Ultra for decoding, targeting 200k-token contexts in GLM-5.3.

    25bookmarks8.8kviews

    AI & LLMs· Discussion

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.