FreeToken: Local Serving of 290B+ MoE Models on Consumer Hardware

RepositoryFlashML-org/FreeTokenTool

FreeToken is an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on desktop PCs and laptops. Using adaptive CPU-GPU co-execution, semantic caching, and dynamic VRAM management, it enables models like DeepSeek-V4-Flash and GLM-5.2 to run locally without datacentre infrastructure.

Key points

  1. Bandwidth-adaptive CPU-GPU co-execution via a q* policy, global LRU expert caching, and custom FTW weight format.
  2. Semantic-aware KV and recurrent state caching avoids redundant computations across agentic tool calls and thinking blocks.
  3. Elastic runtime VRAM re-allocation between expert caches and KV memory without engine restarts.
  4. Support for MXFP4, NVFP4, FP8, and BF16 quantisation formats across NVIDIA RTX 30, 40, and 50 series GPUs.
  5. OpenAI- and Anthropic-compatible API endpoints for immediate integration with coding agents such as Claude Code, Codex, and OpenClaw.

Why it matters

Running frontier-class open-weight MoE models with hundreds of billions of parameters previously demanded enterprise multi-GPU clusters. FreeToken lowers this barrier by unifying GPU, CPU, and host memory on consumer machines. This enables cost-effective, privacy-compliant AI development and autonomous agent workloads directly on local workstations.

Analysis

FreeToken tackles the fundamental challenge of MoE architectures: while active compute per token is low, loading all experts exceeds consumer VRAM limits. By combining prefill streaming, intelligent expert offloading, and runtime memory elasticity, it achieves interactive generation speeds despite PCIe bandwidth constraints. Backed by over 11,000 GitHub stars, an accompanying research paper (arXiv 2608.16157), and ready-to-run desktop apps for Windows and Linux, the project shows solid technical maturity for local AI deployment.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Repository:JustVugg/colibri

    Colibrì: Running Frontier MoE Models via Disk-Streamed Experts

    Colibrì is a lightweight inference engine written in pure C with zero dependencies, designed to run frontier Mixture-of-Experts (MoE) models ranging from 744B to 2.8T parameters on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as a unified memory hierarchy, streaming experts on demand.

    37.7kstarsC

    AI & LLMs· Tool

  • Repository:MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    Serving GLM-5.3 Flash EXL3 on Dual NVIDIA DGX Spark Kits

    Mia's AI Lab provides an optimised vLLM deployment overlay for GLM-5.3-Flash using 4 bpw EXL3 quantisation tailored for 2x NVIDIA GB10 systems.

    431starsPython

    AI & LLMs· Tool

  • Repository:drumih/turbo-fieldfare

    TurboFieldfare: Gemma 4 26B-A4B Inference in 2 GB RAM on Apple Silicon

    TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.

    6.6kstarsSwift

    AI & LLMs· Tool

  • Repository:MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    Running DeepSeek-v4-Flash on a Single DGX Spark

    An open-source launcher and Docker recipe for running DeepSeek V4 Flash 0731 (EXL3) on a single NVIDIA DGX Spark with 128 GiB unified memory.

    328starsPython

    AI & LLMs· Tool

  • Link:drumih

    TurboFieldfare: Running Gemma 4 26B in 2 GB RAM on Apple Silicon

    TurboFieldfare is an open-source inference engine written in Swift and Metal that executes 4-bit Gemma 4 26B-A4B-IT on Apple M-series Macs using roughly 2 GB of RAM. While the model weights require 14 GB of disk space, the engine keeps only the shared model base and KV cache in memory and streams routed expert layers on demand directly from the SSD.

    AI & LLMs· Tool

  • Repository:ashhart/TensorFold

    TensorFold: LLM Inference Engine for Apple Silicon and Modern NVIDIA GPUs

    TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.

    1.1kstarsPython

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.