What's Next After RLHF? Diogo Almeida on Assistance Versus Automation

X postAI Engineer 🔜 Paris 🇫🇷Talk

In his talk at the AI Engineer World's Fair 2026, Diogo Almeida (TypeSafe AI) explores the limitations of Reinforcement Learning from Human Feedback (RLHF), drawing a clear line between human-in-the-loop assistance and dependable automation.

Key points

  1. Both ChatGPT and Claude Code belong to the same paradigm: tools designed to assist an active user rather than operate autonomously.
  2. RLHF optimises for human preference signals, meaning looking right can diverge from being objectively correct.
  3. A core paradox exists: models can solve complex mathematical problems while remaining too unreliable to automate ordinary workflows without supervision.
  4. Assistance requires satisfying the participating user; true automation demands dependable, calibrated decisions running unattended in the background.
  5. Achieving autonomous software requires redesigning the stack around reliability and calibrated decision-making rather than mere conversational fluency.

Why it matters

Teams often struggle when deploying conversational models into autonomous production workflows. Understanding how preference optimisation differs from calibrated correctness is critical for designing dependable, production-ready AI systems.

Analysis

Drawing on his background building GPT-4 and InstructGPT, Almeida provides a grounded critique of prevailing agent narratives. Pre-training and RLHF incentivise models to sound confident rather than communicate uncertainty. Until evaluation and training objectives shift from user satisfaction to calibrated reliability, models will remain powerful assistants rather than self-sufficient software components.

Original post

AI Engineer 🔜 Paris 🇫🇷

@aiDotEngineer · 16 September 2026

full transcript t.co/NJzRWO5i2Y

0 likes0 replies2 bookmarks812 views

View on X
More posts in the thread (1)
  1. congrats to Diogo on a full launch! for more on typesafe: check his aie talk: t.co/pPg9t1cIk4 out now! t.co/rdgkq1qWeC t.co/jfM4eBBMwL
Selected replies (4)
  • @softech28 @aiDotEngineer Interesting 🧠🧠🧠
  • @fde_spot @aiDotEngineer Congrats to Diogo — typesafe tooling is exactly the kind of “ships past the demo” infrastructure AI eng needs.
  • @NealFrazierTech @aiDotEngineer Type-safe agent tooling seems most valuable when it narrows the gap between a prototype and a system you can actually maintain. Releasing with explicit interfaces, observable failure modes, and a clear human handoff gives teams a path to trust automation without pretending it is
  • @NickGideo @aiDotEngineer my favorite talk from the conference!

Mentioned links

Links and tools from this entry

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:Diogo Almeida

    What's Next After RLHF? Why Reliable Automation Demands a Different Objective

    Speaking at the AI Engineer World's Fair 2026, Diogo Almeida (formerly on GPT-4 and InstructGPT, now at TypeSafe AI) argues that RLHF successfully created the assistant era, but falls short for unattended automation. Because RLHF optimises for human preference, models lean towards sounding convincing rather than being reliably correct. Achieving dependable background automation requires shifting towards calibrated, verifiable decision-making.

    AI & LLMs· Talk

  • Article:Wikipedia contributors

    Reinforcement Learning from Human Feedback (RLHF): Principles and Applications

    Reinforcement learning from human feedback (RLHF) is a machine learning method designed to align intelligent agents with human preferences. Instead of manually specifying hardcoded reward functions, human annotators rank model responses. A reward model is trained on these comparisons to guide policy optimisation using algorithms like Proximal Policy Optimization.

    AI & LLMs· Resource

  • Article:Anthropic

    Automated researchers can reliably mitigate alignment failures

    Anthropic demonstrates that AI models can autonomously conduct alignment research. In multi-turn experiments, Claude significantly closed safety gaps across ten alignment failure categories without degrading baseline capabilities. Notably, the weaker Claude Sonnet 5 successfully post-trained an unaligned checkpoint of the more capable Claude Opus 4.8.

    AI & LLMs· Research

  • X post:Mahax

    Moving Beyond Prompts to Loops and Graphs in AI Systems

    Citing Andrew Ng, Mahax highlights the shift from single-shot prompting to agentic loops and graphs. An accompanying guide outlines the architecture of AI agents, self-evaluating feedback loops, and graph-based execution pipelines using Claude.

    11.8kbookmarks1mviews

    AI & LLMs· Resource

  • Article:Mehdi Allahyari

    The Harness Beats the Model: Demystifying Agent Harness Engineering

    Following prompt and context engineering, harness engineering marks the third wave in developing autonomous AI systems. Mehdi Allahyari explains why the software infrastructure wrapping an LLM matters more than raw model scale for reliability and self-correction.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.