Stop Writing Code: Building Software Factories with Autonomous Agents

I wanted to talk a little bit about how the state of software engineering has kind of evolved over the past couple of years. And with the new models like Opus 5.5 and Astra, we're seeing that these models can run for very, very long periods of uninterrupted time and get a lot of work done without you having to even re-prompt or even babysit it. So the first thing I want to talk about is how do we used to do software engineering, and then I'll kind of pivot to like, what I'm currently seeing in the industry now. So previously, you'd have engineers, you'd have UX designers, you'd have a product owner. And the way software kind of evolves is you have a product owner who is basically sitting in front of users and the team to collect requirements and try to understand what to build next. Like what do the users need next, what actually makes sense. And they also prioritize a backlog of tasks. So you'll have this huge backlog of tons and tons of things that people think that they want added into the system. And the product owner's job was to basically get through that list and try to, you know, order it, try to make it make sense. And then once we decide what exactly is going to bring the highest value to users, the product owner will basically tell the team, which is user experience, the designers, the engineers, the QA people, "Okay, now build it." Okay, now there could be some nuance to this. Like some teams would have the task first go to a UX designer who builds a Figma design and gives that over to the engineers to actually code it up. Sometimes you'll have a team where you actually work directly with the UX people and you kind of iterate on the design together, maybe in like a Zoom call or in meetings. But overall there was kind of a pipeline where you'd have the feature request from users, you have the product owner who really understands the core the business domain, who can add those features. And then you have like the engineers, the system engineers, the UI engineers, the backend engineers, who figure out how to get it all done. And this would take a lot of time, right? You'd have one task, would have to go through this whole pipeline. Eventually, you get it deployed somewhere. And then once you get it deployed out there, you realize that you have to fix something that you didn't think about. There's edge cases and gaps that you missed. Sometimes the UX designer person will see the stuff that you added isn't exactly what they wanted because the mockup changed since they made it, or vice versa. There's a lot of in between and overhead that goes on with the traditional software engineering life cycle. Now, let's fast forward to today with AI-enabled companies, and what does that actually look like? And again, I can just give you my experience where what I'm seeing with the things that we work on, and also what I'm seeing on my side projects. So now that we have AI, a lot of the times what we're doing is, instead of interacting directly with each other all the time, all of us are basically interacting with what you call a software factory. Okay? So I'm going to say software factory. Granted, there should still be communication between you guys, like you should have a collaborative experience when you're working with people, otherwise you're not going to build a good product. So you should still have this kind of going on, but instead of it being a pipeline of, "Okay, now the product owner needs X," but what I'm seeing in the industry today is that you can actually have the product owner prompt your software factory, and your software factory can spin up some type of sandbox. That sandbox can run in an isolated environment so it's safe. It can take the prompt from the product owner, it can do the work, Opus can run non-stop for hours until it fully finishes it, and then eventually, the sandbox will make a pull request, which might look like this. So this is an example pull request that I do on my side project for my game, but this exact description is what we do at work as well. Except we have a little bit more detail, right? When you're working on a real product that has real customers and there's real risk, there's a lot more things that you want to show in the description, including how does the architecture the system change? What are the risks involved? Did you do some type of security analysis on it? What's the performance metrics of this change? Does it make anything slower or impact external third party services that might be connecting to you, okay? But in this scenario, it's just a pull request and it just has screenshots of the pumpkin model that I added into this game. Technically, I could add in more about metrics, because I'm adding in a new 3D model. There are triangles involved, which means that this could perform a little bit slower if I'm running this on a MacBook or I'm running this on a GPU, right? There are real impacts for adding a new models to the game, and I could improve this PR description skill that I use in my project to like, do more analysis so that me, the human in the loop, I can just at the very end of the software factory, I can just look at it and be like, "This seems good." Now granted, you can also do some more stuff that have the software factory automatically merge in these pull requests based on, you know the the criteria that you're looking for, but I'll talk about in just a second, okay? So anyway, we've seen at our company, I mean, there's product owners who can now just get features added into a codebase without having to touch or talk to a designer or an engineer. There are designers that I work with who are literally making pull requests right now and changing how UIs work, and changing the design of UIs, and changing the experience of different pages. And they just prompt the software factory, and it goes off, it does a sandbox, it makes a change and makes the pull request, so that the engineers can come through and like review it. Okay? So let's just say pull request is like the the outside of the software factory. Really, the humans are living on the outside now. If you were to think about humans in the loop, humans in the loop, HIL. HIL's human in the loop. So we live in the inputs of the software factory. We have to be the ones prompting it. We have to be the ones gathering requirements and understanding what the user needs. And then we also live on the outside of reviewing the code, reviewing the screenshots, reviewing the metrics. How does this actually impact our system? And we review it and then we get it merged in. Okay? Uh, and then of course, like the UX and the designers could look at the pull request screenshots as well. The product owner could also look at these uh screenshots as well. We just found that GitHub and pull requests are like one of the main ways to kind of do this. And then also, this is the part that's kind of up for debate, depending on your team, in your company, in your organization, realistically, most of these people are just running cloud code on their machines, right? They all run it on their own machines, and the software factory is basically just a collection of skills and workflows and MCPs that everyone kind of runs ad hoc. But in the future, I could see that most companies and organizations no longer even need to have stuff installed on their computer. They can just keep on prompting this cloud software factory environment. It knows how to spin up the sandboxes, it stores all the context, it has the knowledge base. Okay? So, knowledge base, uh it has all the skills already in the repo, and our job now is no longer just getting work done. It's also trying to fine tune this machine, this black box, we're trying to fine tune the skills, we're fine tuning the knowledge base, where we have a feedback loop set up inside the software factory so that every time it's adding in features, it can try to improve the existing skills. Like if you watch Matt Pokock, he has a retretro skill that is supposed to improve the existing skills to make them more accurate. And if you watch the outputs of LLMs that run in these sandboxes, whether that's cloud code or Astra or who knows what. You can have a skill to automatically review the the history logs or the transaction logs of that cloud code session, to make it then make another pull request, that go and fix the skills to make them a little bit better, and this will go and improve the skills, and then at some point you have a pull request that would go and update the skills to make them more accurate. Okay? So we basically live here on the left and at the bottom, and we also have another responsibility. Instead of like writing the code and trying to create the mockups by hand in Figma, we let AI do all that, and we come back through and we iterate on it. And it might take many, many different iterations. It might take a lot of iterations until you actually get this whole process fine-tuned, especially if for your own company and your project, your software factory is going to look very much different than another company. You can't just reuse the exact same skills and apply to every single company in the world. You end up needing to have a team of people who know how to manage and maintain and refine these skills so that every time someone makes changes to this, it updates the proper API docs. It updates the actual ReadMe docs or whatever engineering docs you may have. And then you also need to make sure that you have things in place that are constantly looking at the knowledge base that has business context and other things that are discussed in team meetings, or in Slack chat. These are things that you can just basically put into your knowledge base. Or you can have the sandbox have access to MCPs. So like if we were to say Slack, we can say like Microsoft Teams, all these external services where there's discussions happening, where there's people actually communicating. These could be pulled in for context using MCPs, so your sandbox can more accurately build out and design what needs to happen next. Microsoft, and again, you can hook this up to your Google Cloud Docs. You can hook it up to anything. Basically, you have a bunch of context now that lives outside of your actual software factory, and you need to find a way to hook it in so that every time the AI iterates, it can be more accurate and it can improve on the things that it has. And this is not a pie in the sky idea. Now that Opus 5.5 is out, like this is probably the default standard that a lot of companies are going to slowly transition into. Uh, the ones who are more risk averse will probably take a lot of time to get to this environment or this setup, but I see the future of software engineering, everyone basically logging into one centralized box, they can kick off prompts, they can upload their own images, all their documentation is stored there. But the way software works right now is that we have a bunch of ad-hoc places that we store PDFs and CSV files, and those ad-hoc systems that exist everywhere, tons of different databases and tables. So it gets a little bit more complicated and it's not just throw AI at it. There are actual things we need to consider when it comes to security and risk. Right? For example, having a sandbox have access to all your production databases and all your Slack chats and all your Microsoft Team chats, and all of your meeting transcriptions, maybe that's kind of risky. That could be private information that shouldn't just be thrown in the cloud code. Because again, these models do access a third-party API. You do have Anthropic over here, who is scooping up all of our data, and if you were to push in some sensitive discussion into that, Anthropic may train on that and now other people can have access to it. So, although you may scroll through social media and people will talk about software factories and like, you just need to loop it and set up your loops and your your graphs. This is what I'm seeing right now and I think also like what you need to do as well. Now, if you were to ask yourself as an engineer or like a designer, like where do we play a role in now? Again, it's we have to review the the work that's being done. I will say there's something I didn't mention is that the software factory can have cron jobs or agents that are on constantly running. I'll just say cron right here but I mean whatever trigger that you want to add in, basically there's triggers that happen, whether that's a human manually sending the trigger in or that's Slack sending the trigger in, or that's some type of thing that happened externally to your system is triggering the software factory to do more work. Actually, I'll just say triggers, right? One of the triggers could be every morning you want the software factory to go, review your codebase and try to find performance things it can improve, so improve performance, improve security, uh clean up code, uh update docs. These can all just be cron jobs that run daily, and then when they're done doing their whole sweep through of your code base, whether you have one agent doing that or you fan it out to 10 different agents doing all this work and orchestrating talking to each other, and then finally you're going to get pull requests. Now, let's take it one step further. How do you automate this even more? A lot of the times a pull request, you can just merge it in. You don't need to review it. Another example, I'm working on this uh Jack Lantern like I mentioned, right? If I had some type of gate keep rule that says, "Hey, you can merge in any new pull request as long as it's certified not to reduce the FPS of my game by more than one or two FPS, or the the render loop cannot increase in milliseconds from like 0.01, okay?" If you have an agent that has that specified in context or in some type of script, you can have a trigger that automatically just does all this stuff. It just keeps on cleaning up your code, it could also have a trigger that automatically just looks at your pull requests. Every time a pull request is created, it can automatically review it, and it can automatically merge it back in. So that we are left with humans who only need to focus on the things that are actually core that can change the user experience. All the other stuff behind the scenes, like if it's a high risk change, maybe you're, you know, your agent is doing some work, it's decided to change how your API works or changed the security of your API. That's a very high risk change and you do want a human to review that, okay? So whatever you do, you probably need to make sure that you have in your PR description some type of risk that tells the agents, if you do decide to go this far in terms of automation, this is a low risk PR, it doesn't impact the performance of my game, it doesn't impact the load times of my game, because every time you add a model it has to load in, a user has to load in like more kilobytes to even get past the loading screen, okay? So as long as it's low risk and it doesn't really impact the game, like just merge it. Like what are you waiting for? Why does a human have to come in here and even review this? If the images look good, I should be able to review it. Now I would say that I don't even manually review pull requests anymore. I literally will just prompt an agent and say, "Hey, look at GitHub uh pull request 706, or 702, fix any conflicts that exist on it, address any PR comments that people might have left on it, merge it in when it's done." And I do that for all of them. In fact, I have you typically have a cron job that's just going through and it's just reviewing this stuff. But I will say that's mainly for side projects. For real life work projects, I still try to go through the pull request and review the code, but at some point the human will be the bottleneck, and you and your organization, your team have to decide how many more tokens do we want to throw at the pull request to truly make sure it's secure, to truly make sure it's not impacting users. Do we trust just merging this without a human actually reviewing the code? And that is the fundamental problem that I think a lot of people have to start discussing, is that we know that we can move extremely fast with these tools. With all the different security breaches that are happening on all these different cloud companies, like Railway was one, um I think there's Open code also got hacked. People trying to do distillation attacks. We do have to find that middle ground of like, how fast do we want to move, how much do we want humans to review, and is the workflow, something I didn't mention here is that it's it's not just passing your prompt to the Opus and saying, "Good luck, build it," typically you have a skill that has many different workflows in it, right? You'll have like a planning skill, you have like an implementation skill, you have a different code review skill, you can have different sub-agents reviewing all this at the same time, and then finally it's going to make the pull request. So again, your job is to make sure that this whole pipeline, this workflow, when it gets to your software factory is solid, so that you have less worry about just merging a pull request when it gets created. You have high confidence that it's good, and you can just merge it. Okay? So I think that's kind of like where I'm seeing the industry go, and this is not a pie in the sky idea, like I'm literally doing this at work, and I know other people at other companies are doing this. It's just a matter of when is your company or your team going to actually start implementing something like this, so that all these different people they're not bottleneck by communicating with each other, they communicate with the software factory, and it stores the context of your discussion so it can better craft what you're trying to build. I think the last thing I want to mention is that one of the hardest parts once we have this software factory pretty good, is understanding the business requirements. This chatting with your users and understanding what they want is still like the most important piece, because you can have AI just blindly keep on adding in features and improving various things here and there, but at some point if you don't truly understand the users who are using your system, and you don't understand the existing systems that are in place and what are the risk involved with the other systems? Like you might be on a larger company that has many different teams and they'll manage their own different products. Having AI just unleash and have access to all this stuff at once, probably not a good idea, and maybe your responsibilities have grown a little bit because AI is making you more effective and more productive, but still there are walls that kind of are put up to help insulate the risk. Okay? So you still need to understand, you know the systems, uh the external systems that your system is integrating with. So these are the two things that I have found to become extremely the the main focus now. If you 100% understand what your user needs are, and you 100% how the external systems you're integrating with fully work, and again you can use AI to help get that information, but I found that if you just keep on throwing as much context as possible to Opus, it still gets a little bit confusing and a little bit lost. And so the hard part now, and this has always been hard, it hasn't never been hard, is understanding the business requirements so that you can properly prompt and properly set up the context and your knowledge base so that every single sandbox that's running Opus is going to be that much more successful. I hope you guys enjoy this talk. I wanted to make a little walk-through of the software factory term that we're hearing today. Um but yeah, let me know if there's any other discussions you want me to have. Have a good day, and happy prompting.