Link:Rogerio Bonatti, Dan Zhao, Francesco Bonacci, et al.
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Researchers from Microsoft, Carnegie Mellon University, and Columbia University have launched Windows Agent Arena (WAA), an open-source evaluation framework for multi-modal AI agents operating in native Windows 11 environments. Featuring over 150 real-world tasks, WAA supports parallel execution in Azure to shorten evaluation cycles from days to around 20 minutes. A newly introduced reference agent, Navi, attained a 19.5% success rate against human performance of 74.5%.