Vollständiges Transkript anzeigen (4.682 Wörter)
You've likely found yourself in a situation where you've watched your AI skill nail a task one time, so you figured that it was good to go. But here's the thing. That one win proves almost nothing. If you're new here, I'm Dylan. I run an AI consultancy, and knowing whether your AI skills or projects are actually any good is most of what I do. So, I'll show you how to really test a skill against your own work, then run the loop that takes it from okay to as good as you are. So, by the end of this video, you'll be able to approve it's good, not just hope that it is. So, let's get into it.
I want you to take a second and think about when you hire a new employee or somebody new joins your team. How do you actually go about training them? You likely follow a process something like this, where you give them a small task. After you've given them the small task, you then check their output on that task. How did they do? After you've checked it, you then correct them, and they iterate on this. So, they then try again on the same task or a different task of similar size and complexity. And after you've iterated multiple times, and they've built trust in the fact that they can do this task without you having to review their work, you then hand it off to them, so they can do it completely without your involvement. And the question I tend to ask myself, as well as with my clients, is why do we treat AI differently? It should be the exact same, especially for tasks we want to outsource to AI that are of high criticality or something we do often.
Because often, what I see people do is when you interact with AI, this is how they tend to use it. Where they set up an AI, either a project or a skill, they use it, and then they maybe make one or two minor tweaks, and then going forward, they just accept the fact that the AI is somewhat okay but not perfect. Instead of what they should do is train the AI, specifically that skill or project they've created. They need to iterate with it to meet their standards continuously, and that's exactly what I want to do in this video with you. So, I want to show you how you can take that mindset of training an employee and apply it to AI, so any skill or project you create going forward, meets your standards continuously.
And this is the basic process we're going to follow. I'm going to talk about this in the context of skills, but the same idea applies to system prompts inside of a projects. So, when building a skill, this is the same process I recommend all my clients follow, which is when you build a skill, you prove it first. What that means is you first choose a task that you want to automate with a skill or augment with a skill. Once you've chosen that task is something that you do on a repeated basis, you then do that task with an AI. So, you open up a new chat, either inside of Claude, Co-work, codex, ChatGPT, or Claude, and you run that task live in that conversation. You keep working on that task over and over refining the output from the AI until it meets your standards. Once you've done that in that one conversation flow, you then ask the AI to encapsulate that entire process into a skill for you. There are many reasons as to why this specific approach is the most effective to build skills and system prompts, but the primary one is that the AI has done the work, and the work itself is very nuanced and subtle. So, we need to make sure that it understands all of that through the process of doing it before it creates a system prompt or a skill.
Quick pause in your regular programming. This video is brought to you by me, as always. Two quick things. First off, blows a 30-day AI insight series, completely free. You will get 30 insights in your inbox if I can apply AI to your business and your work. The second thing is if you'd like to work with me, blowers, series of offerings to see if there's a good fit between the two of us. Now, let's get back to the video.
And there's actually a prompt that I'll share with you here that you can copy and paste at the very end of that conversation. And that's this prompt here. It's very straightforward, simple, but important. And in this prompt, all we're doing is telling the AI, I want you to turn what we just did in this conversation into a skill using your skill creator. And that's one of the beautiful things about skills is AI can create them for you. But we do add some constraints to this prompt, which is important. So, the two constraints we're adding here, so the first one is we're telling the AI, I want you to keep this skill lean and minimal. Every line inside of the skill must earn its place. This is critical because, as you probably know, AI tends to be very verbose. It talks a lot, which means it's likely going to create a really bloated skill or prompt. And we want to avoid bloated skills and prompts. Reason being is that the more bloated a skill or prompt is, the less likely the AI is going to follow the task that we gave it accurately. That's the first constraint. The second constraint, especially for skills, is making sure it's agnostic of topic. What we specifically tell it is to capture the judgment and nuance we used for the process, so when we create the skill, it can generally be applied to anything going forward. In this case, we're creating of a proposal, so when I say, this is a skill for a proposal is writing proposals for us, and in this prompt, we're saying, I want you to make sure it generally applies to all proposals of this kind, not just the one that we just worked on in this conversation.
To simply convey that second constraint, what I'm basically stating is that we just built this proposal and I'm happy with the output. Now, I want you to encapsulate the entire process into a skill, so anytime I ask you to write a proposal in the future, it meets these same standards without being biased towards this specific proposal itself and the content inside of it. So, that's the prompt you're going to copy and paste and use to build your skills. Now, at this point, we've done what most people do. We've built a skill or a system prompt, no matter the approach you've taken. The next phase is where a lot of people fall short, and they don't train their AI, but instead, just use it. So, focus up, this is the most important part of the video.
Now, some people may test their skill or prompt, but the way they test it is very poor. These are the two approaches I see people most commonly do. The first approach is they simply read the output from the AI, they do somewhat of a vibe check and say, okay, it's it's kind of okay, it's good enough so I'm going to move on. They're kind of eyeballing the output. That's not great because it's not systematic. The other approach is they try to test it, they open up a new conversation, they call that skill or system prompt with the same input they built it with. And this is also a very bad approach because if you use the same example you built the skill with to test it on, there's a good chance the AI has saved some of that nuance into its native memory. So, it's likely going to give you back something perfect, but if you try it on a different example, it might not necessarily meet your standards. So, we can't test it on the same thing we built it with. And that's where the next approach comes into play. And this is how I recommend all my clients to test their skills and system prompts, especially those that are critical or there's an important task or something you do a lot.
And when doing this testing approach, we need to create two piles of examples. One pile is going to be your build pile. So, this is the pile of examples you use to build the actual skill or system prompt. The other pile is your test pile. This is where you're testing the AI's quality without it cheating. So, you need to make sure it doesn't see this test pile at all when it's going through the building phase, you need to hide this away from it. So, we'll start with the building process. So, the building pile, when you're building the skill, you have the input data and the output data. So, when you're building any type of system prompt or skill, no matter what the task is, the AI is going to take in some data, and it's going to give you back something. Either it's going to give you back some insights, a decision, an output format of a report, a report, a presentation, whatever else. The AI needs to see both of these in the build phase. It needs to see both what you look at and also what you then could produced. Because by seeing both of these, it can meet your standards and your quality expectations. That's the first phase. The second phase of this when you actually start testing it is where only going to give the AI the input because we're assuming there is no output because the AI is going to create it for you. So, we give it just the input, have the AI run the skill, it then gives us the output. Now, it's important that we test this on things we've already done manually. So, we can then compare the AI's output versus our output and determine how close the AI is to our standard for that specific example. So, once it gives us the output, we then compare the two. Now, this is where, again, a lot of people fall short.
When we're doing the comparison between the AI's output and our output, it needs to be objective, and the criteria we're evaluating the AI on needs to be binary. And that brings us to the next slide, which is creating that criteria. And the AI, luckily, can help us with this process. So, this prompt is something you're going to copy and paste and use yourself. You're going to use this prompt after you've created the skill. So, after you've created the skill, then you need to have the criteria to embed in that skill. So, this is going to be a fresh conversation you start with the AI. And in this prompt, we're simply telling the AI, I want you to create a checklist that tells me if my drafted proposal is good. So, in this case, you're going to swap this out for whatever task you're doing. So, I'm doing it on a drafted proposal, you could be doing it on something else. After this, we then give the AI three samples. So, I'm saying, here are three samples of different proposals I've written and have won effectively. Again, this is going to be three samples of whatever you're applying this to. After that, we then say, and then here's the skill that's going to draft new proposals in the future. So, the AI has access to this skill, and also the three samples you've written yourself that are good. It's going to look at both of these, it's going to look at the proposals first. And when it looks at the proposals, it's going to systematically review them and figure out what makes these good. After it's determined what makes these proposals good, it's going to distill that down into, ideally, a list of 7 to 10 questions that are binary. So, there's a yes-no answer to it. And it has to be binary. Reason being is that we're going to have the AI grade its own work as it writes a proposal for you. We'll come to that later on. And that's really all this prompt is doing. So, once you have your criteria back and we're happy with them, it ideally looks something like this. So, these are binary questions that are related to the proposal example they're walking through.
So, here you can see, we're stating in the first question that does the proposal start with the client problem, yes, no. After that, we ask is the price on the first page, and then is the proposal under two pages. Every single one of these questions are binary, yes, no, pass, fail. Once we have our criteria back and we're happy with them, we then want the AI to then embed those criteria into the skill. And I'm actually going to give you a prompt that does that for you, but first, I'm going to say there's a very specific way we want it to embed it into the skill and use it. And that's what this visual represents. So, let's say that we've created a skill that writes a proposal for us. So, the parent AI, so the AI we're interacting with is going to write the proposal. After it's written the proposal, it's going to spawn or create a sub-agent or a baby AI. That baby AI is going to act as a grader. It's going to grade its own work against the criteria we just created. The reason that we're going through the process of creating a sub-agent instead of having the parent AI judge its own work is that if we had the parent AI judge its own work, it's biased towards that. So, it's likely going to say, yes, everything is passed because I'm amazing. We don't want that. We don't want the AI to be biased. So, by having the AI spawn a sub-agent or a baby AI, that new AI has its own context window and it has less bias towards the output. So, it then does the grading against the checklist, and provides the parent AI back a report of its findings. After it's got the report, then that parent AI sends back the final report as well as a drafted proposal. Now, let me show you a simple prompt that you can copy and paste that does this for us. So, this is the prompt that we're going to add to the AI once we have our criteria and our skill. And it's a very simple prompt. All we're saying is I want you to create a grading step inside of our existing skill that we've already created. Within this grading step, after the skill writes the proposal, so this is again my use case, so you'll change this to whatever you're using it for. I want you to create a separate sub-agent that acts as a grader that's going to grade this with fresh eyes, looking only through the finished proposal as well as the yes-no checklist that we provide to you. I want you to then provide back a report of your findings that shows each question, the answer of yes-no that it passed or not, and then if it didn't pass, I want you to give a one-line reasoning or proof as to why it didn't. And this is going to be really important here. So, this specific segment here of getting the reason as to why it didn't pass is going to help us kick off the next step of this process, which is the loop, and that's improving the skill over time to get as close to our quality as possible. And that's all this prompt does. That leads us to the next phase, which is the loop. So, right now we have our skill created with a criteria inside of it, and we've added the additional instructions so the AI grades its own work with a sub-agent. Now, once it's graded the work and it's given us the output back, we need to see did it check off all the boxes, yes, or are there some no's. If there are some no's, we need to improve the skill. And this is the prompt that's going to help you do that. So, once you have your output, you're going to copy and paste this prompt into that same thread that you got the grade the report out with as well as the proposal. So, what we do here is we give the AI our written proposal that we wrote, so this again, this is for the blind test. So, we ideally have a sample of proposals we've written in the past, and we're testing the AI's capability of writing that same proposal with the same inputs. So, we're saying, here's the proposal that the skill wrote, the AI already has access to this. We then give it the proposal that we wrote manually ourselves, so this is new information for it, as well as the checklist of questions that failed, which again, this is the AI already has this in the conversation. So, we're saying, we're saying here are the three things, you have the proposal the AI wrote, the proposal I wrote, and the associated checklist of no's of where the AI failed. Now, we're asking the AI, I want you to methodically look at the specific skill instructions and figure out exactly where the skill failed us and why the AI didn't get close enough to our specific standards for this type of proposal. Once you've identified the areas where it's failed, I want you to make the smallest possible changes to the skill to fix this systematic issue that's likely going to repeat for future proposals, not just this one. And after you've made that tiny change, you then show me exactly what you changed. Now, a few things here. The reason that we're saying I want you to make the smallest possible change goes back to the point that I made previously. AI tends to be very wordy and we don't want it to bloat our skill with these changes. So, we need to make sure that the changes targeted and minimal. In addition to that, we want the AI to tells exactly what it changed so we're aware of the improvements and if it made a change that we disagree with, we can have it revert that. And this is the process. So, you've created the skill, you've then tested it with blind data sets, and then you ran that loop continuously. You're likely going to have to do this multiple times until the AI meets your standards. But even after the AI is met your standards and it's working continuously at the expectations you have, there's a few reasons as to why you might want to run this process again. And that's what this represents. So, over time, new models get released, and every time a new model gets released, you want to test that new model on your specific process to see if it gets embedded, better, or worse. And if it does it worse, what's likely happening here is that there's too much of a prompt around the smart AI, you need to whittle that prompt down. And this is something I've talked about in many other videos in the past, but the TLDR here is that when you have a really intelligent model that gets released and you have a huge prompt for an older, dimmer model, that huge prompt usually acts as constraints on the AI, holding it back from its true power. So, you sometimes have to revisit these. Another reason you would revisit this process is that your standards may change for that process. Maybe they've risen and gotten even higher. And if that's the case, you need to update your process running the loop and improving the skill as well as the associated criteria, so the AI can get better as well.
So, let's do a quick recap of the most important things we talked about, specifically three things. First, when you're using AI, don't simply use it, train it like you would train an employee, and we've already walked through the process of how to do that. One of the key parts of that process is proving that the skill or system prompt is good on your use case, and the only way you can do that is through blind testing. So, you test it on a data set that is not seen and you only give it the inputs and they compare it against your previous outputs to see how close it is. And then over time, after you have your evaluation criteria that are binary, you'll run that loop continuously. And every time that an AI comes out and it has a series of no's instead of yeses, those are the areas that you can improve, that's the targeted fix you can make. And that's where the loop kicks off, and that's how you improve the skill in the system prompt. And that's all this prompt does. That leads us to the next phase which is the loop. So, right now we have our skill created with a criteria inside of it and we've added the additional instructions so the AI grades its own work with a sub-agent. Now, once it's graded the work and it's given us the output back, we need to see did it check off all the boxes, yes, or are there some no's? If there are some no's, we need to improve the skill. And this is the prompt that's going to help you do that. So, once you have your output, you're going to copy and paste this prompt into that same thread that you got the grade the report out with as well as the proposal. So, what we do here is we give the AI our written proposal that we wrote, so this again, this is for the blind test, so we ideally have a sample of proposals we've written in the past, and we're testing the AI's capability of writing that same proposal with the same inputs. So, we're saying, here's the proposal that the skill wrote, the AI already has access to this. We then give it the proposal that we wrote manually ourselves, so this is new information for it, as well as the checklist of questions that failed, which again, this is the AI already has this in the conversation. So, we're saying we're saying here are the three things: you have the proposal the AI wrote, the proposal I wrote, and the associated checklist of no's of where the AI failed. Now, we're asking the AI, I want you to methodically look at the specific skill instructions and figure out exactly where the skill failed us and why the AI didn't get close enough to our specific standards for this type of proposal. Once you've identified the areas where it's failed, I want you to make the smallest possible changes to the skill to fix this systematic issue that's likely going to repeat for future proposals, not just this one. And after you've made that tiny change, you then show me exactly what you changed. Now, a few things here. The reason that we're saying I want you to make the smallest possible change goes back to the point that I made previously. AI tends to be very wordy and we don't want it to bloat our skill with these changes. So, we need to make sure that the change is targeted and minimal. In addition to that, we want the AI to tells exactly what it changed so we're aware of the improvements and if it made a change that we disagree with, we can have it revert that. And this is the process. So, you've created the skill, you've then tested it with blind data sets, and then you ran that loop continuously. You're likely going to have to do this multiple times until the AI meets your standards. But even after the AI has met your standards and it's working continuously at the expectations you have, there's a few reasons as to why you might want to run this process again, and that's what this represents. So, over time, new models get released, and every time a new model gets released, you want to test that new model on your specific process to see if it gets better, or worse. And if it does it worse, what's likely happening here is that there's too much of a prompt around the smart AI, you need to whittle that prompt down. And this is something I've talked about in many other videos in the past, but the TLDR here is that when you have a really intelligent model that gets released and you have a huge prompt for an older, dimmer model, that huge prompt usually acts as constraints on the AI, holding it back from its true power. So, you sometimes have to revisit these. Another reason you would revisit this process is that your standards may change for that process. Maybe they've risen and gotten even higher. And if that's the case, you need to update your process running the loop and improving the skill as well as the associated criteria, so the AI can get better as well.
So, let's do a quick recap of the most important things we talked about, specifically three things. First, when you're using AI, don't simply use it, train it like you would train an employee, and we've already walked through the process of how to do that. One of the key parts of that process is proving that the skill or system prompt is good on your use case, and the only way you can do that is through blind testing. So, you test it on a data set that is not seen and you only give it the inputs and then compare it against your previous outputs to see how close it is. And then over time, after you have your evaluation criteria that are binary, you'll run that loop continuously. And every time that an AI comes out and it has a series of no's instead of yeses, those are the areas that you can improve, that's the targeted fix you can make. And that's where the loop kicks off, and that's how you improve the skill in the system prompt. And that's all this prompt does.
And that's it. So, as a reminder, two quick things. First off, blows a 30-day AI insight series, completely free. You'll get 30 insights in your inbox if I can apply AI to your business and your work. The second thing is if you'd like to work with me, blowers, series of offerings to see if there's a good fit between the two of us. Okay. So, before you go, it's important to pick the right skill before you train it. This loop only pays off when the skill already fits your work. A skill you build carries your context. A downloaded skill somebody else built doesn't. So, the first call you have to make with any skill is knowing if you're going to build, borrow, or clone it. I'll walk you through all three right here in this video. Go ahead and give it a click. I'll see you next time, Internet.
Video
Video
Video
Video
Video
