Modular MAX: Open-Source Framework for AI Serving and Modeling Across Heterogeneous Hardware

LinkModularTool

Modular has introduced MAX (version 26.5), an open-source AI serving and modeling framework designed for CPUs, GPUs, and ASICs. The framework eliminates CUDA and PyTorch dependencies, relying on Mojo for portable GPU kernels.

Key points

  1. Cross-platform execution across NVIDIA, AMD, and Apple silicon without requiring matched CUDA or ROCm installations.
  2. OpenAI-compatible inference endpoints with out-of-the-box support for models including DeepSeek, Gemma, and Qwen.
  3. PyTorch-like Python API for model authoring alongside low-level GPU kernel development written in Mojo.
  4. Reported throughput of 171% compared to vLLM (tested with Gemma3-27B on AMD-MI355x: 15.557 QPS versus 9.091 QPS for vLLM 0.10.1).
  5. Smaller container images and faster cold start times due to zero dependencies on PyTorch, CUDA, or ROCm; package distribution via uv, pixi, or Docker.

Why it matters

Removing reliance on proprietary hardware runtimes lowers barrier-to-entry for alternative accelerators, helping teams avoid vendor lock-in while streamlining containerised deployment workflows.

Analysis

MAX addresses deep fragmentation across AI computing platforms. By pairing a custom graph compiler with Mojo-based kernel implementations, Modular abstracts away low-level vendor toolchains. The emphasis on AMD MI355x performance underscores an industry push to provide viable, high-throughput deployment alternatives to existing CUDA-centric stacks.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:Modular

    Modular: Unified AI Inference Stack from Kernel to Cloud

    Modular presents its AI inference platform designed to execute workloads portably across hardware architectures including NVIDIA, AMD, TPUs, Trainium, and Apple silicon, combining the MAX serving framework with the Mojo language.

    AI & LLMs· Tool

  • Link:mojolang.org

    Mojo Releases Version 1.0.0 and Goes Fully Open Source

    The Mojo programming language has reached version 1.0.0 and is now fully open source under the Apache License 2.0 on GitHub. Mojo targets systems programming for AI-era heterogeneous hardware, blending high performance with Python interoperability.

    AI & LLMs· Resource

  • Repository:ashhart/TensorFold

    TensorFold: LLM Inference Engine for Apple Silicon and Modern NVIDIA GPUs

    TensorFold is an open-source inference engine designed to serve large language models on Apple Silicon (via MLX/Metal) and newer NVIDIA GPUs (Compute Capability 8.9+). It provides OpenAI- and Anthropic-compatible endpoints with model-tailored kernels and speculative draft verification.

    1.1kstarsPython

    AI & LLMs· Tool

  • Repository:LMCache/LMCache

    LMCache: Scalable KV Cache Management Layer for LLM Inference

    LMCache is an Apache-2.0 open-source caching layer designed to accelerate LLM serving by optimising and sharing Key-Value caches across processes and network nodes on Nvidia and AMD hardware.

    11.7kstarsPython

    AI & LLMs· Tool

  • Link:Donato Capitella

    Strix Halo AI Toolboxes: Containerised Environments and Host Tuning for AMD Ryzen AI MAX+

    Donato Capitella provides Strix Halo AI Toolboxes, a collection of containerised environments and host configuration guides for AMD's Ryzen AI MAX+ platform. Leveraging unified memory, the hardware allows the integrated GPU to address up to 128 GB of RAM for local LLMs, image generation, and model training.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.