Testing your AI assistant: what to watch for and what to do with what you find
One of my recent articles walked through building an AI assistant around a real workflow - choosing the task, defining the role, drafting instructions, adding context files and guardrails, and running a quick sanity check. If you followed that process, you now have something concrete to work with.
This article is about what comes next: testing that assistant properly, so you can see where it genuinely helps and where it still needs work.
TL;DR
Testing an AI assistant means using it on real work and paying attention to how much effort you are still putting in to get usable results. The moments where you have to steer, clarify, correct, prompt again, or compensate for missing guidance are the signal - they show you exactly what to tighten so the next run takes less intervention. Run the assistant through two or three realistic scenarios, vary the inputs, and keep a running note of what happens. Then organise your findings into three groups: what worked, what broke, and what felt rough. Separate the symptom from the likely cause, and you have a concrete brief for improving the assistant rather than guessing at what needs changing.
Why one good run is not enough
AI can behave differently depending on the input, the wording, and the context it receives, so a response that works well with one set of inputs might drift or fall apart with another. One successful run, even a genuinely impressive one, only tells you that the assistant worked in that specific situation, with that specific input, on that specific day. It does not yet tell you whether the assistant will behave reliably across the range of situations you actually need it to handle.
This matters because the natural temptation after building an assistant is to test it once, feel good about the result, and start relying on it. But if the assistant has only been tested on your best-case scenario, you do not yet know what happens when the brief is vague, the inputs are incomplete, or the task is slightly different from the one you had in mind when you wrote the instructions.
Working through the assistant in enough varied scenarios is what gives you the evidence to see where it holds up and where it still needs work. The aim is to see its real behaviour across the kinds of situations you will actually use it in, rather than its best-day behaviour on a perfect input.
What you are actually looking for
Testing is about how smooth the human-AI interaction feels when you use the assistant to finish a real piece of work.
You should run it like you are doing the job for real - following its flow, answering the questions it asks, and using its output to get the deliverable to a finished state. As you go, you need to pay attention to every point where the interaction becomes effortful: where you have to explain the task again, correct a wrong assumption, fight the output format, or repeat yourself to get it to take the next sensible step.
Those friction points are where the useful information lives. They represent the hidden effort sitting between the assistant producing just something and the assistant producing something you can actually use. A five-hour manual task might come down to one hour with the assistant, but if that hour is spent prompting it back on track and compensating, there is still meaningful improvement to be made, and the friction points you noticed during the run show you exactly where to focus.
Where friction tends to show up
When you are working through the assistant, there are a few specific areas worth watching.
The opening. When the assistant starts, you want it to make clear what it is going to help with and what it needs from you. The key is whether you can get oriented quickly and start the work, without first having to figure out what the assistant is actually trying to do. This also ensures the assistant is truly anchored in its purpose.
The questions it asks. The questions need to be specific enough that you can answer them without guessing. A vague question like "can you provide information about the client?" pushes the work back onto you, because you end up deciding what counts as relevant information and how much to provide. A clearer version - "Can you share the client's website URL, plus any relevant docs like a brief, notes, or an old proposal, so the assistant can understand who this is for?" - gives you something specific to respond to and keeps the assistant doing the work it is meant to do.
The pattern of questioning also matters, and the right approach depends on the workflow. Some workflows are best with the same core questions every time, because predictability is what keeps things stable, while others need the questions to adapt based on earlier answers so the conversation moves forward instead of looping back over things already covered. Whichever pattern fits the workflow you are building, the questions should feel clear and purposeful when you are using them.
How it handles incomplete input. When you give a short or vague answer, you want the assistant to notice and ask a follow-up rather than carrying on with what it has and producing something generic. In practice, people often provide less detail than the assistant expects, often because they do not know what it needs. A well-structured assistant notices when the input is thin and asks for more before doing the job.
The output itself. The output should be usable with at most light tidying, match the tone you defined in the knowledge files, and follow the structure you laid out in the instructions. The detail that matters more than people realise is whether reviewing the output feels like a quality check, rather than a full rebuild before you can trust it.
The overall experience. The interaction should feel easy to follow, with a clear sense at each stage of what is happening and what comes next. After a session, you should be able to look back and see clear forward progress on the work, without lots of small moments where you had to nudge the assistant back on track.
How to run a useful test
The simplest approach is to pick a real scenario and work through the assistant from start to finish, as if you are relying on it to get a deliverable done.
Use a realistic example, including any messiness it normally comes with - a brief from a real project, a real situation you would actually be planning for, or whatever scenario the assistant is built for. The point is to see how the assistant handles the kind of input it will actually receive in day-to-day use.
As you work through it, keep a running note of what happens. A simple list is enough. For each friction point, write down what happened, where it happened, and why it felt off, with something like: "Asked me for 'project details' but didn't specify what kind of details it needed, so I wasn't sure whether to include budget, timeline, or just the scope."
Once you have been through one scenario, run a second one with different inputs - a different kind of brief, or different answers to its questions. Two or three varied runs will show you whether the assistant behaves consistently across real use cases.
If you have access to someone who has not used the assistant before - a colleague, a team member, someone who could realistically use it - asking them to try it is one of the most useful things you can do. You will learn more from watching someone else use it for five minutes than from testing it yourself for an hour, because they will hit friction points you have unconsciously learned to work around having built the assistant yourself.
Organising what you find
After a few runs, you should have a list of observations. The next step is to sort them so they are useful when you sit down to improve the assistant.
Group your findings into three areas.
What worked well. Worth recording because it tells you which parts of the assistant are solid and should be left alone when you start making changes. It is tempting to rework everything once you are in improvement mode, but if something is doing its job, protect it.
Where the assistant broke or drifted. The clear failures - wrong output, missed steps, ignored instructions, off-brand tone, or anything that would leave the user stuck or confused.
Where the experience felt rough. Smaller moments where the assistant could have been clearer, smoother, or more helpful - a question that was technically answerable but awkwardly phrased, an output that was correct but in the wrong format, or a transition between steps that felt abrupt.
One thing that helps when you reach the improvement stage is separating what went wrong from why it went wrong. A generic output is the symptom you can see, but the cause sits underneath - the assistant might not have asked enough questions before it started drafting, a knowledge file might not be connected to the right step in the instructions, or the rules might not include guidance on the level of specificity you expect.
When you record a friction point, try to note what you think is behind it. You will not always get it right, and that is fine, but thinking about causes now will save you time later. The more specific you can be - something like "the output was wrong because the assistant didn't ask about the audience before drafting" - the easier it will be to make a targeted fix when you come back to improve the assistant.
What testing actually gives you
Testing is what builds your confidence that the assistant will support your workflow reliably, with less effort from you each time.
The observations you capture become the brief for your next round of improvements, so when you sit down to make changes to the assistant you have something concrete to work from.
In a future article, I will go through how to turn those findings into targeted improvements to the instructions and knowledge files, so the assistant performs more consistently with less steering from you.
If you are building AI assistants for your business and want to understand how to test and improve them properly - or you have built some already and the outputs are not consistent enough - I am happy to talk it through.
