pi-bench: Benchmarking Local Coding LLMs on AMD Hardware

LinkDonato CapitellaResource

The pi-bench project evaluates the pi-coding-agent on local AMD hardware, specifically Strix Halo and Radeon 9700 GPUs. It aims to help developers gauge the real-world coding performance of current models and quantizations running locally.

Key points

  1. Dedicated to local AMD hardware: Specifically targets AMD Strix Halo and Radeon 9700 GPUs.
  2. Evaluated via SWEBench-verified-mini: Utilises a 50-datapoint curated subset preserving original difficulty and pass rate distributions.
  3. Hybrid evaluation: Newer runs combine automated containerised test execution with Gemini 3.1 Pro serving as an LLM judge.
  4. Default configuration: Runs bare pi-coding-agent without custom system prompts, external tools, or custom wrappers.
  5. Extensive benchmark runs: Evaluating a single model can take upwards of 24 hours.

Why it matters

Developers aiming to run autonomous coding workflows locally need clear metrics on consumer- or workstation-tier AMD hardware to decide between cloud APIs and local setups.

Analysis

By focusing on Strix Halo and Radeon GPUs, pi-bench offers valuable data points outside the usual Nvidia ecosystem. The use of a standard agent setup ensures tests reflect raw model and quantisation performance rather than prompt engineering.

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Similar in content, found by the AI search.

Let's talk about your project

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.