LiteRT-LM: Google's Open-Source Inference Framework for On-Device LLMs

Repositorygoogle-ai-edge/LiteRT-LMTool

LiteRT-LM is an open-source inference orchestration layer from Google designed to run large language models directly on edge devices. It leverages GPU and NPU acceleration and powers on-device generative AI features in Chrome, Pixel Watch, and Chromebook Plus.

Key points

  1. Cross-platform support across Android, iOS, Web, desktop, and IoT devices such as Raspberry Pi.
  2. Hardware acceleration targeting GPUs and NPUs, including speculative decoding support.
  3. Broad model support including Gemma (e.g. Gemma4-E4B), Llama, Phi-4, and Qwen.
  4. Multi-modal input support (vision and audio) alongside tool use and function calling for agentic workflows.
  5. Release v0.16.0 introduces prebuilt versioned C API libraries for simplified native integration and an experimental YNNPACK delegate on Linux ARM64.

Why it matters

On-device inference lowers latency, removes cloud server expenses, and strengthens privacy by keeping sensitive data on the user's hardware. LiteRT-LM provides the necessary tools and prebuilt binaries to integrate local LLMs smoothly across heterogeneous device ecosystems.

Analysis

LiteRT-LM addresses the friction of deploying compute-intensive models to resource-constrained client hardware. Implemented in C++ around LiteRT and XNNPACK/YNNPACK, it offers bindings and interfaces for CLI, Python, JavaScript, and C. The framework targets mobile, desktop, web, and embedded engineers building local AI capabilities. Despite the v0.16.0 version numbering, the project demonstrates high maturity through active production usage across Google's own products like Chrome and Pixel Watch.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Repository:ggml-org/llama.cpp

    llama.cpp: LLM Inference in C/C++

    llama.cpp is an open-source C/C++ project by ggml-org dedicated to LLM inference. Boasting over 127,000 GitHub stars, it provides pre-built binaries, Docker containers, REST APIs, and C library bindings for executing language models.

    127.1kstarsC++

    AI & LLMs· Tool

  • Repository:drumih/turbo-fieldfare

    TurboFieldfare: Gemma 4 26B-A4B Inference in 2 GB RAM on Apple Silicon

    TurboFieldfare is a dedicated Swift and Metal runtime designed to run Google's Gemma 4 26B-A4B on Apple Silicon Macs within an approximately 2 GB RAM budget. It avoids loading the full 14.3 GB model into memory by keeping only the 1.35 GB core and KV cache in RAM while streaming the required mixture-of-experts weights from SSD on demand.

    6.6kstarsSwift

    AI & LLMs· Tool

  • Repository:ashhart/TensorFold

    TensorFold: LLM Inference Engine for Apple Silicon and Modern NVIDIA GPUs

    TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.

    1.1kstarsPython

    AI & LLMs· Tool

  • Repository:JustVugg/colibri

    Colibrì: Running Frontier MoE Models via Disk-Streamed Experts

    Colibrì is a lightweight inference engine written in pure C with zero dependencies, designed to run frontier Mixture-of-Experts (MoE) models ranging from 744B to 2.8T parameters on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as a unified memory hierarchy, streaming experts on demand.

    37.7kstarsC

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.