Show the full transcript (2,381 words)
My name is Suraj Gupta. I lead harness development at Warp. Um, in case you haven't heard of Warp before, um, we started off by building a state-of-the-art agentic development environment. Um, we have about a million active users that use that. Um, it was born out of a modern-day terminal. Um, and nowadays we're building a, um, cloud agent platform to enable teams to build software factories. Um, today I'm going to talk about, um, self-improving software factories, which I know might sound kind of buzzy. Um, but I'm going to break it down and hopefully you can walk out of here with something concrete that you can implement yourselves.
Um, you've probably heard of both of these terms, self-improvement and software factories, used throughout this conference, but I'm sure you've heard of them independently. Um, you've heard about how self-improvement is about agents being able to, um, improve over time, agents and models, um, and, you know, phasing out humans from that loop so that way the the the improvement loop can be automatic. Um, and I'm sure you've heard about software factories being, um, the way that, like, the the software development paradigm is shifting from individual coding agents to building out automations that take work from triage up until production.
Um, but less has been said about how software factories themselves can self-improve over time so that the the factory can get better, more efficient, faster. Um, and I think this is going to become an extremely important part of our jobs as software engineers as we transition from building products to building factories and, and maintaining them.
Um, so let's dive in. Um, self-improvement is a pretty broad umbrella, but I'm going to focus on three concrete ways that agents can improve over time, agent factories can improve over time. I'm going to start with, um, skills, then we'll talk about persistent memory, and then I'll finish off by talking about model routing.
Um, so, skills are a great way to give your agents procedural memory. I don't think I have to kind of spell that out for you. You describe a, a procedure to your agent that you expect it to have to do a task over and over. Um, for example, let's say you're building a triage agent, which is a pretty common thing you might build in a software factory. Um, and you give it a skill on how you want it to go about reproducing issues.
Um, those skills get stale over time. Um, humans get feedback to the agents, agents learn things in their own, um, runs and trajectories, um, and the the the skill that you previously gave it becomes dated. So how do we fix this and let the agent improve over time using that skill?
Well, one way to do it is to deploy an outer loop agent. Um, and this is what we've done in our own internal software factory that's been really productive. The idea is that your, um, your inner loop agent is the thing that's actually applying the skill. It's like doing the triage. And then you have this other outer loop agent that is observing your inner loop agent's runs and improving its skill over time by looking for, um, for mistakes it made or looking at feedback that the human has given it.
Um, so I'm going to show an example of that. Um, let me swap here. Cool. Um, so this is actually a skill that we've used internally at Warp. We've open-sourced our client repository and it's fully run as a as a factory. And so this is an example of an of a of a skill that's backing a triage agent that we have today. The idea is that, um, you you get an issue that comes in on GitHub and you want to automatically run an agent to figure out like what sorts of, um, like it what sort of information do the user not provide that is going to help us figure out if this is something that we should build, an existing issue that we might actually just want to dedupe against, or something else.
Um, and then we have a, sorry, let me just switch over. And then we have a workflow that actually like runs that that triage, um, agent. Um, this is a pretty simple workflow. It just every time an e an an issue gets submitted on the repo, this, um, this workflow runs. And then here's the actual outer loop agent, which is about looking at the the work that the triage agent has done and improving it over time.
So the idea is that the agent looks at the decisions that the triage, the inner loop triage agent made and it, um, collects feedback signals, which might be, uh, you know, thumbs up or thumbs down on the issues, um, it might be users commenting on the issue, might be someone from Warp, an employee of Warp commenting on the issue, uh, giving the agent feedback on what it did.
Um, and then ultimately that leads to like some synthesis of of that, of that iteration. And so we, we end up basically updating our inner loop skill with this outer loop agent. Um, and that's what leads to the self-improvement loop with skills. What's really nice about this is that, um, the so in step four, you'll see we, we edit the triage skill and then we open a pull request. So that means that, um, all of the improvements to the inner loop skill are going to be tracked through git, so you get like full observability into like how that skill is transformed over time. And that it also means that, um, a human is actually going to review those updates to the skill. So that way, the the this outer loop agent doesn't make a mistake and ultimately cause your triage agent to to actually perform worse.
Um, yeah, so that's that's kind of what what I want to talk about for skills. Um, let me talk about persistent memory now. So I'm going to go back to my deck. Um, cool. So skills are great for remembering procedures, but what about everything else? So like, what, what happens when your century agent, um, looks for an issue and, for an issue and finds an issue, does some work to like gather context about it, and fixes it, but then now you you encounter a similar issue in the future? Um, your century, your agent might get lucky and like be able to figure out the same root cause that it did the first time around. But that's not guaranteed. And even and even if it does, you've probably wasted a lot of tokens, regathering context on an issue that you've already fixed before.
So that's where, um, something like persistent memory comes in. You can think of this as a fact store scoped to an agent. Um, your agent runs and similar to the skill improvement loop, you have an outer loop agent that's going to extract facts, learnings, outcomes, um, from your, from your inner loop agent. So that way future runs of the agent can lean on what's already been done in the past.
Um, so I'm going to show an example of this. I'm back to the deck here. So in Warp, um, or in Oz rather, uh like I mentioned, we built out a a cloud agent platform where you can run agents, um, in the cloud. And, uh, for all for all these agents, you can attach memory stores to them. Again, a memory a memory store is a collection of facts. Um, in this case I have a Sentry agent that, um, has collected some memories, uh over time about, uh root cause analysis that it's done in the past.
What's cool is like you can actually version this memory, you can update it as a human, create new memories or delete them yourselves. Um, but primarily this this memory creation loop is driven by agents. Um, you can also see where these memories were sourced from. So, um, I can actually like open up the run here, um, to see, oh, well, I think this my IP address here keeps changing and this my our staging platform is gated to an IP address, but anyways, um, you should like you can see where those memories were sourced from, so that way you can be like, oh, like maybe that wasn't actually an important memory or that was kind of like a local maxima that I don't want to include in my memory store.
Um, and then when your agents continue to run, so in this case I have the Sentry agent that's, um, that's done a bunch of runs after it's created those memories, it actually can lean on those memories in the future, um, when it's making, um, when it's triaging new issues. So in this case, you'll notice that, um, this agent ran, it did a bunch of work to do some root cause analysis, and then it ultimately reported back on which memories it used, um, uh, in this case it it was able to use those, um, five memories that I had in that memory store to to speed up the work that it did this time around.
Um, cool. Uh let me go back to my deck. So, what's cool in Oz is that persistent memory works across all harnesses. So, Warp's own proprietary harness, CloudCode, Codex, it doesn't really matter what you're running on Oz, um, a memories will automatically be created for you. Um, but you also have the control to, to review those memories, version them, delete them or edit them as you please. Um, all while being fully traceable.
Um, cool. It's what I wanted to say about memory. Uh last thing I want to talk about is model routing. So, um, you might be wondering how this plays into self-improvement loops for, um, factories. Well, first I want to touch on why model routing is important in a factory in the first place. Um, it can become prohibitively expensive to run all of your agents that are doing simple things like triage or fixing simple CI failures with Opus. That's just like going to be really expensive over time and I'm sure some of you have maybe been burnt by this or have heard stories of, of organizations being burnt by this.
Um, so model, like choosing which models you use for which tasks, that is actually very important, um, and something that we've been working on at Warp. Um, so in Warp, you can actually use, uh our out-of-the-box model routers. We call them our auto models. Um, over like as new models come out, we're all we're always evaluating like what are the best models when it comes to Pareto efficiency. And so like, if you're a new user and you just want to get started, it's much better than like pinning your model to like Opus or or or Haiku, for example.
Um, let me show you all, um, that experience in Warp. So, let me go here. Give me one sec, got to drag this over here. So here I have my Warp desktop, um, and you can see that, um, alongside being able to define models, um, or being able to use Warp's auto models as one option, like these models here. But you can also define your own model rooted, um, uh, models, which, uh, basically boil down to like a set of rules that you can give, um, to us and we will like make decisions about model routing based on those rules.
So here I have an, an example of a configuration file that I've defined, um, where I've defined a few different rules about, like, I want database migrations done with GLM fire, with GLM, or I want, um, runbooks and API documentation done with Quen, um, so this really allows you to like look, to define classes of tasks and be able to say, okay, I want this, this class of task done by this model because that's kind of what you've seen.
At this point, though, that's kind of more of an art than a science, um, and so, uh, what we're working on next is being able to define Evals for, um, customer-facing Evals where you can actually define what sorts of things you care about and what knobs you want to turn on these configurations, and actually see, okay, is, is Haiku better than, than GLM, than Opus, in a specific class of issues that's like specific to your workflows and not just some like generic benchmark that, um, that someone posted online.
Um, let me go back here, give me a sec. Cool. Um, and yeah, so the, let me, uh, oh, I'm, I'm on my slideshow. So here it's kind of like a, a high-level overview of how this works in Warp today. So, um, internally we have our model routing rules, um, but then we have like an eval sidecar that's determining, um, what sorts of tasks are actually better with which models, and we do this with a simple, like Best-at-K, um, approach where when you type a prompt, we run a bunch of agents in Oz, uh, across the different set of models.
Um, and we found success in like being able to determine, oh, like, UI tasks are really well done with GLM. Um, we don't really need to run those with Opus. It's much more efficient to run those with GLM. Um, so that's how we've been using it, but, um, sooner enough we're going to be building this into our product so that way our our customers can actually use it in their own workflows.
Um, yeah, that's, uh, that's mostly what I had today. Um, we're around in a booth around that side, so if you're interested in this kind of stuff, come talk to me. I'm always down to talk about this kind of stuff. Yeah, thanks for your time today. That's all I had.
Video
Video
