LMCache: Scalable KV Cache Management Layer for LLM Inference

RepositoryLMCache/LMCacheTool

LMCache is an Apache-2.0 open-source caching layer designed to accelerate LLM serving by optimising and sharing Key-Value caches across processes and network nodes on Nvidia and AMD hardware.

Key points

  1. Acts as a dedicated KV cache layer for LLM inference runtimes such as vLLM and PyTorch.
  2. Supports Nvidia CUDA as well as AMD ROCm architectures, including AMD MI300X accelerators.
  3. Features production-ready multi-node P2P CPU memory sharing for distributed caching.
  4. Introduces a multiprocess architecture designed to improve performance for agentic workloads and Mixture of Experts (MoE) models.
  5. Member project of the PyTorch Foundation.

Why it matters

Long-context and multi-turn agentic workloads consume substantial GPU memory through KV caches. Offloading and sharing KV caches across CPU memory and nodes reduces latency and infrastructure costs when hosting generative models at scale.

Analysis

With more than 11,000 GitHub stars, adoption within the PyTorch Foundation, and demonstrated use alongside frameworks like vLLM, LMCache presents a mature solution for teams managing high-throughput, self-hosted LLM infrastructure.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Repository:ashhart/TensorFold

    TensorFold: LLM Inference Engine for Apple Silicon and Modern NVIDIA GPUs

    TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.

    1.1kstarsPython

    AI & LLMs· Tool

  • X post:0xSero

    How KV Caching Works and Its Trade-offs in LLM Inference

    A technical breakdown by Avi Chawla, shared by 0xSero, explains the mechanics of Key-Value (KV) caching in transformer architectures. The method eliminates redundant computations during autoregressive token generation, providing a roughly 5x speedup in practice while shifting the bottleneck to GPU memory.

    343bookmarks36.4kviews

    AI & LLMs· Tutorial

  • Repository:eugr/spark-vllm-docker

    spark-vllm-docker: Optimised vLLM Docker Setups for DGX Spark

    spark-vllm-docker is an open-source tool providing Docker configurations and startup scripts to run vLLM on DGX Spark systems, supporting both single-node setups and multi-node clusters via Ray or PyTorch distributed mode.

    2.2kstarsShell

    AI & LLMs· Tool

  • Repository:hhhuang/CAG

    Cache-Augmented Generation (CAG): A KV-Cache Alternative to RAG

    The CAG open-source repository introduces Cache-Augmented Generation as an alternative to Retrieval-Augmented Generation (RAG). By preloading the entire knowledge base into the context window and reusing the precomputed KV cache, it bypasses real-time document retrieval entirely.

    1.5kstarsPython

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.