Local AI Registry: Validated GPU Recipes for Local LLM Deployments

Repositorysybil-solutions/local-ai-registryResource

The Local AI Registry provides reproducible configurations for running open-source LLMs on specific GPUs. Every recipe is pinned to exact model commits and container engine digests, backed by evidence from six functional checks and a minimum threshold of 15 tokens per second.

Key points

  1. Recipes must pass six gates: load, chat, reasoning, tool calling, context window retention, and at least 15 tokens per second.
  2. Guarantees reproducibility by pinning Hugging Face model commits and exact container engine image digests.
  3. The compiled catalog (catalog.json) offers at most three curated recipe picks per GPU configuration.
  4. Includes Python and JS SDKs, a Linux CLI runner, and integrations like the Omarchy Local AI plugin.

Why it matters

Engineers avoid tedious trial and error with quantisation and context parameters by relying on benchmarked, proven configurations tailored to their hardware.

Analysis

Deploying local LLMs reliably requires extensive testing across quantisations, context windows, and serving engines (such as TabbyAPI) on specific GPU hardware. The Local AI Registry tackles this friction by introducing an evidence-based configuration repository.

Architecture and Workflow

Recipes are structured as lightweight JSON files organised by GPU vendor and model. Each recipe ties pinned Hugging Face weights to an engine profile defined by an immutable image digest. Deviations from default engine parameters, context window targets, and a validation proof containing throughput metrics (TPS and prefill speeds) are tracked directly in the metadata. If an issue arises, launches can be marked as withdrawn with an audit trail.

Maturity and Tooling

The project provides a complete workflow: the lab.py utility validates and renders container commands, static APIs are published via Cloudflare Pages, and SDKs in JavaScript and Python facilitate programmatic lookups. With around 280 stars and a structured gating system, the repository offers a mature, practical foundation for local inference deployments on Linux.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 2 November 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Repository:eugr/spark-vllm-docker

    spark-vllm-docker: Optimised vLLM Docker Setups for DGX Spark

    spark-vllm-docker is an open-source tool providing Docker configurations and startup scripts to run vLLM on DGX Spark systems, supporting both single-node setups and multi-node clusters via Ray or PyTorch distributed mode.

    2.2kstarsShell

    AI & LLMs· Tool

  • Repository:syv-ai/qwen38-27b-rtx3090

    Qwen3.8-27B on a Single RTX 3090: High-Throughput Serving with vLLM

    This repository provides an optimised serving configuration for running Qwen3.8-27B on a single 24 GB consumer GPU (RTX 3090) using vLLM 0.28.0 and contexts ranging from 150k to 262k tokens.

    1.1kstarsPython

    AI & LLMs· Tool

  • Link:spark-arena.com

    Spark Arena: LLM Performance Leaderboard on NVIDIA DGX Spark

    Spark Arena is a public leaderboard tracking Large Language Model performance on NVIDIA DGX Spark hardware. Powered by open-source tooling, it provides transparent benchmark comparisons across inference engines including vLLM, SGLang, TensorRT-LLM, and llama.cpp.

    AI & LLMs· Tool

  • Repository:MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

    One-Click Installer for Qwen3.8-27B on 16 GB Nvidia GPUs

    A serving kit for Windows and Linux enables running the Qwen3.8-27B language model locally on consumer Nvidia GPUs with 12 to 16 GB of VRAM using EXL3 quantisation.

    529starsPython

    AI & LLMs· Tool

  • Repository:ashhart/TensorFold

    TensorFold: LLM Inference Engine for Apple Silicon and Modern NVIDIA GPUs

    TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.

    1.1kstarsPython

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.