LinkdrumihTool
Running larger models locally on consumer laptops is typically blocked by memory constraints on 8 GB or 16 GB machines. Streaming MoE expert weights dynamically from fast NVMe storage overcomes RAM bottlenecks and makes capable 26B models accessible on entry-level hardware.
Because Mixture-of-Experts (MoE) architectures route individual tokens to specific subsets of weights, storing every parameter in memory is not strictly required. TurboFieldfare bridges the latency gap between SSD and RAM by hiding parallel pread reads behind the GPU execution of shared layers, supported by an expert cache. Achieving 5–6 tokens per second on an 8 GB M2 Air and exceeding 30 tokens per second on an M5 Mac demonstrates that selective disk streaming is a viable deployment strategy for local edge AI.
Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Repository:drumih/turbo-fieldfare
TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.
6.6kstarsSwift
Repository:JustVugg/colibri
Colibrì is a lightweight inference engine written in pure C with zero dependencies, designed to run frontier Mixture-of-Experts (MoE) models ranging from 744B to 2.8T parameters on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as a unified memory hierarchy, streaming experts on demand.
37.7kstarsC
Repository:ashhart/TensorFold
TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.
1.1kstarsPython
Repository:FlashML-org/FreeToken
FreeToken is an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on desktop PCs and laptops. Using adaptive CPU-GPU co-execution, semantic caching, and dynamic VRAM management, it enables models like DeepSeek-V4-Flash and GLM-5.2 to run locally without datacentre infrastructure.
11.7kstarsPython
Link:inclusionAI
inclusionAI has introduced Ling-3.0-tiny, a compact MoE model with 7.9B total parameters activating only 1.3B parameters per token. Utilizing a hybrid architecture of linear and latent attention mechanisms, it aims to deliver efficient reasoning and agentic workloads on local hardware.
Repository:PipeNetwork/kimi-k3-mlx
kimi-k3-mlx is an Apple MLX port of Moonshot AI's native multimodal MoE model Kimi-K3 (2.78T total parameters). Because the model exceeds the memory capacity of any single Apple Silicon machine, the repository includes a custom streaming converter, vision components, and expert-pruning tools.
337starsPython
Parts of this content were created with the assistance of AI.
