Show the full transcript (2,072 words)
Today I’m going to talk about these things. They’re DGX Spark servers used for local AI. Specifically, the Gigabyte and ASUS variants. They both use the Grace-Blackwell GB200 chip and are essentially identical to the NVIDIA reference model. Now, I have to admit, the experience of owning these things has been a little bit of a roller coaster ride. I went from being really excited opening the box to almost returning them. But I didn’t do it. And now after using them for more than a month, I have to say I think I kind of love these little boxes.
Before we get started, I want to address the elephant in the room here. This is a polarizing device and people tend to love it or hate it. And the thing is there’s a kernel of truth to some of the frustration. It’s not as turnkey as something like a Mac Studio, and it can lock up if you use too much memory. And by the way, the reason I almost returned mine is because it was locking up. It also takes a lot of technical expertise to get it up and running. And the inference speed is a little slower than a Mac Studio, although that may be changing soon. I’ll explain that in a little bit. But if you can get past the learning curve, there’s something really special going on here. So the purpose of this video is to get you up and running as quickly as possible and hopefully avoid all the hassle and frustration.
Okay, let's start with the hardware, and I want to begin by focusing on the differences between these two. There really aren’t that many. First, let's look at the airflow. The ASUS has the cold air entering from the bottom and exiting out the back. The Gigabyte is straight through from front to back, and I think I kind of prefer this style. One of the weird quirks about the Gigabyte is there’s no LED light, and believe me, I looked. It’s more annoying than you might think. On startup, it’s essentially silent, and sometimes you’re just not sure if it’s running or not.
Other than that, they’re essentially the same. They both have 128 gigabytes of unified memory, 10 gigabit Ethernet, and both of mine are using one terabyte NVMe drives. Also on the back, there are four USB-C connectors with one dedicated to power. Here there’s a HDMI connector and two ConnectX network adapters which can run at 200 gigabits per second. I gotta say, connecting two DGX Sparks is a huge topic and not for the faint of heart. I’ll have a separate video covering that.
Finally, the star of the show is the GB200 superchip. The GB200 is actually two chips called dielets, side-by-side. On this side we have the Blackwell GPU, and on the other side is the Grace CPU. In terms of data flow, all the inputs enter from the CPU side. On the far right is the high-speed I/O which is where all the PCI lanes enter the chip. And on the top and the bottom here, we have the memory interface where the unified memory comes in. In the middle, there’s this bluish strip called the chip-to-chip or C2C interface. It’s a fast 600 gigabyte per second highway that moves data between the GPU and the CPU. Okay, a couple of important things I want to point out. The memory bus is 256 bits wide, and that’s half the width of the Mac M4 Max chip, which is the main reason why the Mac has a faster inference during the decoding phase. Another interesting thing is each PCIe device is limited to four data lanes. It’s not a big deal unless you’re planning to use the ConnectX-7 port, and again, I’ll explain that in a future video.
In terms of power use, it draws about 25 watts at idle and about 50 watts during peak RAM and GPU usage during inference. When rendering images and video, it peaked at about 180 watts. Case temperatures peaked at about 110 degrees Fahrenheit, but usually run pretty cool under most circumstances. Overall, the power usage and temperature ratings are about on par with a Mac Studio, but maybe just a little bit higher.
Alright, now let’s get this thing set up and start serving some models. As always, I’ll have all of my instructions in my GitHub page linked below. And for this video, I’ll assume you have a working knowledge of Ubuntu and I’ve already set up things like timezone and language settings. So at this point, you should be logged in to a terminal prompt. The first thing you’ll do is create a Docker group and add it to your user. The reason that’s important is a lot of these LLM serving engines will run inside of a Docker container. Creating this group will allow you to run Docker without the sudo command. I also installed a few things like Homebrew, UV for Python, and Rust. Some of these are optional, but they’re all nice to have. The next thing you want to do is set up a Python environment by running this command. Now that we have the initial setup out of the way, we have to choose a serving engine, and there are a lot of them. But these are the two approaches that I recommend. The first one is a community repo in Git called Eugar. And I hope I pronounce that right. For serving, it uses VLLM with a custom Docker image and some nice recipes to make things easy. Of all the serving methods I’ve tried, this is without a doubt the most turnkey. Every recipe just seems to work and it has this curated feel to it.
There is one setting that’s really important though. Remember I said I was running into issues with the server locking up? You see when your model uses all the available memory, it gets into a bad state and can cause it to seize up. To fix that, you want to adjust this GPU memory utilization setting. Whenever I run into an issue, I set this to 0.5, then slowly increase it until the problems return. Then I throttle it back just a little bit. This will solve most of the issues when serving a model. The installation is pretty easy. I start by creating a folder to store everything, then I clone the repo. Next, go into the Spark VLLM Docker folder and run the build-and-copy script. This will create a Docker image that’s built for the DGX Spark with all the required libraries. Once the container is built, you’ll have two options for serving. The first option is to run the launch cluster script, and this has the VLLM serve command in line. It works, but my favorite approach is these the recipes. In the recipes directory, you’ll see a list of the available options. All you have to do is execute the run recipe script and then the name of the recipe. And by the way, if you’re running a single spark, you’ll have to use the solo argument or you’ll run into some issues. Okay, after a few minutes you should have a working model. You can test it using a curl command or an agent. Here’s an example where I’m using the Continue plugin for VS Code. Going back to the DGX Spark terminal, you can see things like token per second and some other statistics.
Another great serving option is SparkRun which is related to the Spark Arena community page. I gotta say, the DGX Spark community is amazing. There are about a dozen or so keen individuals who’ve put in a lot of work to make this platform viable. And you can really tell the community’s trying to squeeze every token per second out of these little boxes. So check out the Spark Arena page and especially the leaderboard. To get SparkRun going, you can check out the project at this GitHub page or on their main site. The installation is pretty simple. Just run this UV command and walk through the setup wizard. Like I said, the community’s amazing and you can tell there’s overlap with the Eugar project and SparkRun. For example, the Eugar recipes work with the SparkRun project. So the recipes can be mixed and matched. And lately I find them running the Eugar recipes using the SparkRun runtime. Okay, to run a recipe, you can either click this button to create the command, or you can copy the recipe to a local file and run sparkrun run and then the name of the file.
I think right now one of my favorite recipes is this one. It’s called Qwen 3.6 35B using the Prisma Quants. It’s pretty impressive using a single user session. I was getting roughly 70 tokens per second. But when I run four user requests at the same time, I was getting a peak of 250 tokens per second. Now, keep in mind that’s aggregate between the request. But it’s become apparent to me that running multiple requests is one area where the DGX really shines, especially when you’re using something like VLLM.
Now let’s talk about performance. Compared to the Mac Studio, the inference speeds have been about 20 to 50% slower on the DGX Spark during the decoder phase. So that’s expected given the difference in the memory bus between the two devices. However, while I was working on this video, something unexpected happened. A new model dropped called Diffusion Gemma, and this model is completely flipped the script. Instead of being a memory bound during decoding, it uses a diffusion model that’s kind of sort of similar to image rendering. This uses more CPU and gives the DGX Spark an edge over the Mac hardware. I was getting almost three times the token per second count when using this model. Now I’m not sure if these models are going to end up replacing autoregressive models, but if they do, it could make the GB10 devices a major contender in inference. Another area where the DGX Spark really shines is image and video rendering. This is another instance where the performance numbers flip in favor of the DGX over the Mac. For images and video, I’m using an application called ComfyUI. Again, the installation is pretty easy. Just run this pip command, then run Comfy install. To start ComfyUI, run this Python command. Note that I’m adding a listen argument and that will allow me to access the web app remotely. Next, open up your web browser to port 8188 to bring up the app. An easy way to get started is to click templates and grab a pre-made workflow. Let’s try this one. It’s called Qwen Image 2512. You’ll notice this complaining about some missing models, and this is where things get a little tricky with the DGX Spark. You see on a Mac or PC, when you click download, it automatically places the model in the right location. But since this is a web app on the Spark, the models go into your download folder. So you’ll have to find the appropriate folder under comfy/ComfyUI and then the models folder. For example, we would place this model in the diffusion models folder. Once the models are all downloaded and put in the correct location, you can start rendering an image. To mix things up, I’ll change the text a little bit and make her bag red. Okay, the first render took roughly five minutes. But if I click the turbo option, I can render an image in 20 seconds or so.
Next, let’s select a video template so we can render an image as a video clip. This is a tiny painting that my wife made, and she was going to throw it away, but I thought it looked really cool and would be nice for my studio. So let’s see what happens when we try to bring it to life with video. That is pretty wild. I also thought the time to render was pretty quick for such a compact machine. Okay, in my next video, I’m going to walk through the process of connecting two DGX Spark servers together. Until then, thanks for stopping by, and I’ll see you in the next video.
Video
