Repositoryashhart/TensorFoldTool
TensorFold bridges hardware-level execution across Apple Silicon and the latest NVIDIA hardware with standard client API protocols. The inclusion of dedicated drafting mechanisms helps maximize inference efficiency on high-end local workstations and GPU servers.
The project serves developers and infrastructure engineers looking for high-performance self-hosted inference compatible with popular client libraries. Its technical stack relies on Python, MLX, and modern CUDA math, intentionally excluding legacy GPUs below compute capability 8.9. In terms of maturity, the project is under active development with over 1,000 GitHub stars and an Apache 2.0 license, though some components (such as EXL3 support and select compiled GPU builds) remain experimental.
Summary generated by AI (Gemini 3.8 Flash, 7 October 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Repository:drumih/turbo-fieldfare
TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.
6.6kstarsSwift
Repository:ashhart/MCDMA
MCDMA enables Remote Direct Memory Access (RDMA) across heterogeneous workstations. Built in C under Apache-2.0, it bridges Apple Silicon Macs via Thunderbolt with Linux hosts like NVIDIA DGX Spark and AMD Strix Halo to accelerate distributed AI workloads.
178starsC
Repository:NVIDIA/Model-Optimizer
NVIDIA Model Optimizer (ModelOpt) provides state-of-the-art model optimization techniques such as quantization, pruning, distillation, and speculative decoding. It accepts PyTorch, Hugging Face, and ONNX models and outputs optimized checkpoints ready for downstream engines like TensorRT-LLM, vLLM, and SGLang.
5kstarsPython
Link:drumih
TurboFieldfare is an open-source inference engine written in Swift and Metal that executes 4-bit Gemma 4 26B-A4B-IT on Apple M-series Macs using roughly 2 GB of RAM. While the model weights require 14 GB of disk space, the engine keeps only the shared model base and KV cache in memory and streams routed expert layers on demand directly from the SSD.
Repository:LMCache/LMCache
LMCache is an Apache-2.0 open-source caching layer designed to accelerate LLM serving by optimising and sharing Key-Value caches across processes and network nodes on Nvidia and AMD hardware.
11.7kstarsPython
Repository:FlashML-org/FreeToken
FreeToken is an open-source inference engine designed to run massive Mixture-of-Experts (MoE) models on desktop PCs and laptops. Using adaptive CPU-GPU co-execution, semantic caching, and dynamic VRAM management, it enables models like DeepSeek-V4-Flash and GLM-5.2 to run locally without datacentre infrastructure.
11.7kstarsPython
Parts of this content were created with the assistance of AI.
