Knowledge Base

Link:Diogo Almeida·27 Sept 2026

What's Next After RLHF? Why Reliable Automation Demands a Different Objective

Speaking at the AI Engineer World's Fair 2026, Diogo Almeida (formerly on GPT-4 and InstructGPT, now at TypeSafe AI) argues that RLHF successfully created the assistant era, but falls short for unattended automation. Because RLHF optimises for human preference, models lean towards sounding convincing rather than being reliably correct. Achieving dependable background automation requires shifting towards calibrated, verifiable decision-making.

From X post: What's Next After RLHF? Diogo Almeida on Assistance Versus Automation

X post:Hanako·11 Sept 2026

Evals vs. Loops: Why Post-Hoc Checks Fail to Fix Agent Workflows

Hanako breaks down the limitations of conventional evaluations in AI systems, contrasting them with loop and graph architectures. While evals only report failures after execution completes, loops act as gating mechanisms between individual steps to halt compounding errors.

96bookmarks8.3kviews

Link:Hamel Husain·5 Sept 2026

Hamel Husain’s Blog: Applied AI Engineering and Evals

Machine learning engineer Hamel Husain shares technical essays and guides focused on AI evaluations, LLM tooling, and data science.

From Video: AI Weekly Updates: Open-Source Model Surge, Agent Harnesses, and Meta's AI Reality Check

Article:Kuldeep Paul·5 Sept 2026

Comparison of Five No-Code AI Agent Builders and Evaluation Strategies

A guide by Maxim AI reviews five prominent no-code platforms for building AI agents and highlights why systematic evaluation is critical before deployment. By exposing HTTP endpoints or webhooks, agents built in n8n, Gumloop, MindStudio, Lindy, and Relevance AI can be tested systematically against quality and reliability metrics.

Article:docs.google.com·5 Sept 2026

AIEWF 2026: Build Evals that actually matter

A slide deck from the AIEWF 2026 event titled 'Build Evals that actually matter'. The slide contents were not captured in the extracted text.

From Video: Building Production-Grade Agent Evals: Lessons from Lyft

Video:AI Engineer·19 Jul 2026

Building Production-Grade Agent Evals: Lessons from Lyft

Nick Ung and Akshay Sharma from Lyft present an end-to-end framework for evaluating customer support AI agents. They detail why default offline simulations produce misleading pass rates and show how fine-tuned adversarial user simulators and calibrated LLM judges create trustworthy benchmarks.

Video:AI Engineer·15 Jul 2026

Don't Ship Skills Without Evals: Best Practices from Google DeepMind

Google DeepMind's Philipp Schmid explains why AI agent skills require systematic evaluation suites and how to build lightweight eval harnesses. Drawing on data from SkillsBench and internal practices at Google, he outlines how to write, structure, test, and eventually deprecate skills as foundation models improve.