Repositorymw00/project-mayaTool
It allows engineering teams and enterprises to deploy frontier-scale MoE models fully on-premise, eliminating cloud API costs and ensuring confidential data never leaves private infrastructure.
Project Maya, developed by Peasant Smith (@mw00) alongside co-maintainer @needmorevram, tackles the immense hardware barrier of frontier Mixture-of-Experts (MoE) models. Running models like DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), and GLM-5.3-Flash (321B) previously required datacentre-grade clusters. Maya accomplishes local execution on consumer and workstation hardware using intelligent dynamic expert paging. The CUDA-based engine benchmarks the host system's GPU VRAM, CPU, RAM, and NVMe storage bandwidth, dynamically relocating model experts according to conversational attention patterns. Frequently used experts remain in VRAM, intermediate weights stay in system RAM, and dormant parameters stream directly from NVMe drives. Beyond raw inference, the project ships as a complete operational package featuring a self-hosted server, web dashboard, dedicated quantisation tools, and drop-in compatibility for OpenAI and Anthropic API formats supporting concurrent requests. The codebase exhibits solid early maturity under the MIT licence with over 2,000 GitHub stars and verified GGUF weights on HuggingFace. Potential adopters should note that AMD ROCm and Windows compatibility remain experimental, and sustainable throughput still demands top-tier workstation components with rapid storage buses.
Summary generated by AI (Gemini 3.8 Flash, 28 October 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Repository:FlashML-org/FreeToken
FreeToken is an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on desktop PCs and laptops. Using adaptive CPU-GPU co-execution, semantic caching, and dynamic VRAM management, it enables models like DeepSeek-V4-Flash and GLM-5.2 to run locally without datacentre infrastructure.
11.7kstarsPython
Repository:JustVugg/colibri
Colibrì is a lightweight inference engine written in pure C with zero dependencies, designed to run frontier Mixture-of-Experts (MoE) models ranging from 744B to 2.8T parameters on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as a unified memory hierarchy, streaming experts on demand.
37.7kstarsC
Repository:MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks
Mia's AI Lab provides an optimised vLLM deployment overlay for GLM-5.3-Flash using 4 bpw EXL3 quantisation tailored for 2x NVIDIA GB10 systems.
431starsPython
Repository:ashhart/TensorFold
TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.
1.1kstarsPython
Repository:MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark
An open-source launcher and Docker recipe for running DeepSeek V4 Flash 0731 (EXL3) on a single NVIDIA DGX Spark with 128 GiB unified memory.
328starsPython
Repository:drumih/turbo-fieldfare
TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.
6.6kstarsSwift
Parts of this content were created with the assistance of AI.
