LinkTool
While many local LLM runtimes serve strictly as background inference engines, Inferencer exposes the internal mechanics of model generation, providing observability into token confidence and low-level steering options.
Inferencer targets developers and power users who require privacy and precise control over model execution. Its primary differentiator lies in observability and steering: visualising token entropy highlights generation uncertainty, while prefilling and expert capping let users directly manage response structure and hardware utilisation. With compatibility layers for Ollama, OpenAI APIs, and IDE agents such as Continue.dev and GitHub Copilot, it bridges the gap between raw local inference and structured developer workflows.
Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Repository:FlashML-org/FreeToken
FreeToken is an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on desktop PCs and laptops. Using adaptive CPU-GPU co-execution, semantic caching, and dynamic VRAM management, it enables models like DeepSeek-V4-Flash and GLM-5.2 to run locally without datacentre infrastructure.
11.7kstarsPython
Repository:drumih/turbo-fieldfare
TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.
6.6kstarsSwift
Repository:ashhart/TensorFold
TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.
1.1kstarsPython
Link:drumih
TurboFieldfare is an open-source inference engine written in Swift and Metal that executes 4-bit Gemma 4 26B-A4B-IT on Apple M-series Macs using roughly 2 GB of RAM. While the model weights require 14 GB of disk space, the engine keeps only the shared model base and KV cache in memory and streams routed expert layers on demand directly from the SSD.
Repository:google-ai-edge/LiteRT-LM
LiteRT-LM is an open-source inference orchestration layer from Google designed to run large language models directly on edge devices. It leverages GPU and NPU acceleration and powers on-device generative AI features in Chrome, Pixel Watch, and Chromebook Plus.
6.4kstarsC++
Repository:ggml-org/llama.cpp
llama.cpp is an open-source C/C++ project by ggml-org dedicated to LLM inference. Boasting over 127,000 GitHub stars, it provides pre-built binaries, Docker containers, REST APIs, and C library bindings for executing language models.
127.1kstarsC++
Parts of this content were created with the assistance of AI.
