Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

LinkRogerio Bonatti, Dan Zhao, Francesco Bonacci, et al.Research

Researchers from Microsoft, Carnegie Mellon University, and Columbia University have launched Windows Agent Arena (WAA), an open-source evaluation framework for multi-modal AI agents operating in native Windows 11 environments. Featuring over 150 real-world tasks, WAA supports parallel execution in Azure to shorten evaluation cycles from days to around 20 minutes. A newly introduced reference agent, Navi, attained a 19.5% success rate against human performance of 74.5%.

Key points

  1. WAA adapts the OSWorld framework to offer 154 deterministic tasks across everyday applications, including VS Code, LibreOffice, web browsers, and Windows settings.
  2. The system infrastructure relies on Docker containers hosting Windows 11 VMs connected through Flask servers, with Azure Machine Learning handling parallel test runs in roughly 20 minutes.
  3. The accompanying agent Navi employs chain-of-thought prompting alongside Set-of-Marks representations using UIA tree parsing, DOM trees, OCR, and OmniParser.
  4. Benchmark results indicate that agents relying solely on pixel-based OCR underperform compared to those integrating the Windows UI Automation tree, while human accuracy still significantly exceeds all current models.

Why it matters

Earlier agent benchmarks frequently focused on isolated web navigation, coding, or text-only sandboxes. By testing agents across a full operating system with standard desktop software, WAA provides a realistic testing ground, while its Azure-based parallelisation drastically speeds up iteration cycles for researchers.

Analysis

The benchmark underscores the current limitations of OS-level agents: Navi's 19.5% success rate compared to 74.5% for humans demonstrates that chained multi-step reasoning remains difficult. Crucially, the authors show that relying solely on raw vision and OCR is less reliable than tapping into OS-level accessibility trees like UIA. This suggests that near-future desktop agent architectures will depend heavily on hybrid screen-parsing techniques combining visual models with system accessibility APIs.

Mentioned links

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang

    ToolSandbox: Stateful and Conversational Benchmark for LLM Tool Use

    ToolSandbox is an evaluation benchmark that assesses LLM tool-use capabilities using stateful environments and multi-turn interactive dialogues.

    AI & LLMs· Research

  • Repository:nekuda-ai/WindTunnel

    WindTunnel: Benchmarking Browser Agent Interfaces and WebMCP

    WindTunnel is an open-source benchmark suite designed to compare how browser agents interact with websites. It evaluates WebMCP (direct site-exposed actions) against coordinate-based computer use via screenshots and page structure navigation using DOM and accessibility trees.

    34starsHTML

    AI & LLMs· Research

  • Article:Terry Chen, Yeyin (Eva) Zhu, Zhifan Ye, Jean-Francois Puget and Humphrey Shi

    NVIDIA AVO Achieves 100% on ARC-AGI-3 Benchmark

    NVIDIA has introduced Agentic Variation Operators (AVO), an agent architecture that enables Claude Opus 5 to reach a 100% completion rate on the ARC-AGI-3 interactive reasoning benchmark.

    AI & LLMs· Research

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.