Laya: Non-Autoregressive Decision Engine for Fast In-Text Choices

RepositoryNandhaKishorM/layaTool

Laya is an open-source Python decision engine designed to deliver typed decisions—such as categorical choices, scores, or yes/no answers—across more than 100 languages. Instead of relying on autoregressive text generation, it produces decisions in a single forward pass taking approximately 33 ms.

Key points

  1. Executes typed text decisions (choice, score, yes/no) in a single forward pass in roughly 33 ms.
  2. Supports over 100 languages and contexts up to 8,192 tokens using the laya-multilingual checkpoint.
  3. Trained using reinforcement learning against strictly proper scoring rules (RLCD).
  4. Features an integrated router that selects the appropriate checkpoint per incoming request.
  5. Provides optional extras for ONNX Runtime, TileLang GPU acceleration, LangChain, MCP servers, and HTTP serving.

Why it matters

Many steps in agentic workflows and pipeline routing do not require token-by-token text generation, only fast and reliable classification. Laya avoids the latency and compute overhead of standard generative LLMs for pure decision-making tasks.

Analysis

Laya caters to AI engineers and developers needing ultra-low-latency classification, routing, and guardrails in agent frameworks. Built on Python 3.10+, PyTorch, Hugging Face, and ModernBERT, it offers broad operational tooling from ONNX and TileLang to LangChain and MCP. With over 25,000 GitHub stars, public weights on Hugging Face, and an Apache 2.0 license, the project exhibits a strong level of community adoption and maturity.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:Featherless.ai

    Simple Jev: Open-Source Decision and Classification Engine Using Model Logits

    Featherless.ai has introduced Simple Jev, an open-source implementation that turns open language models into structured decision engines. Instead of requiring a separate classifier head, it reads next-token logits for allowed labels and returns probabilities and choices directly as JSON.

    AI & LLMs· Tool

  • Repository:google-ai-edge/LiteRT-LM

    LiteRT-LM: Google's Open-Source Inference Framework for On-Device LLMs

    LiteRT-LM is an open-source inference orchestration layer from Google designed to run large language models directly on edge devices. It leverages GPU and NPU acceleration and powers on-device generative AI features in Chrome, Pixel Watch, and Chromebook Plus.

    6.4kstarsC++

    AI & LLMs· Tool

  • Repository:LMCache/LMCache

    LMCache: Scalable KV Cache Management Layer for LLM Inference

    LMCache is an Apache-2.0 open-source caching layer designed to accelerate LLM serving by optimising and sharing Key-Value caches across processes and network nodes on Nvidia and AMD hardware.

    11.7kstarsPython

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.