An AI assistant is easier to evaluate when it has a specific job. “Help staff find approved guidance” is a workable starting point. “Transform the organisation” is not a testable brief. Choose a bounded task, define the review process and judge the pilot against examples the team understands.
Pick a task with a clear boundary
Look for a repeatable information task that consumes time and has an understandable outcome. It might involve finding a passage in approved documents, preparing a first draft or comparing two versions of a document. Write down what the assistant may do and where a person must take over.
An internal guidance assistant, for example, could help a staff member locate relevant information. It should not quietly become the authority that decides policy or eligibility. That distinction belongs in the brief and in the way the experience is presented to users.
Prepare the material it will use
Review the source documents before introducing the assistant. Remove superseded versions, identify the current owner and make conflicts visible. A tool cannot resolve a policy disagreement simply because both versions are available to it.
Choose a manageable source set for the pilot. Include ordinary questions, ambiguous wording and cases where the answer is not present. The team needs to see how the assistant behaves when information is missing, not only when the correct response appears clearly in the first document it reads.
Test the answer you do not want it to guess
An evaluation set should expose the limits of an assistant before people rely on it in ordinary work.
Tick a check as you review it.
Keep expected answers and review reasons with the test set so changes can be assessed consistently.
Create a review set before the demonstration
Collect representative tasks from the people who will use the assistant. For each, describe what a useful answer should contain, which source supports it and what would make it unacceptable. Include questions that require the assistant to say it cannot determine the answer.
Review the output for accuracy, completeness and practical usefulness. A fluent answer can still omit a condition that changes its meaning. Keep the assessment understandable enough that subject-matter reviewers can explain why an answer passed or failed, rather than relying on a single unexplained score.
Make human review part of the experience
Users should understand whether they are reading a draft, a summary or a pointer to an authoritative source. Give them a practical way to check the information and report a problem. Decide who reviews those reports and how corrections reach the next version.
For a drafting assistant, approval may sit with the person responsible for the final document. For a document-search tool, the user may need to read the original passage before acting. Different tasks need different review arrangements; one general disclaimer does not design that work.
Use the pilot to make a decision
Compare the pilot with the agreed tasks and observe how staff actually use it. Look at errors and review effort as well as convenience. A useful finding might be that a narrower task works well while a broader one needs better source material.
The deliverables are a use-case brief, a working pilot, an evaluation set and a clear recommendation. Expansion should follow evidence from that review. The team should also be able to decide that the task needs a simpler tool or a better information process instead.
Explore our Practical AI assistants and evaluation, related services and Build a dashboard people can check and use.
Questions and answers
Can an AI assistant make eligibility or policy decisions?
A search or drafting pilot should not quietly acquire that role. Define the decision boundaries, approval responsibilities and escalation route before use, with appropriate governance for the particular service.
What makes a good first AI project?
Choose a repeated information task with current source material, a clear owner and an observable result. A narrow pilot is easier to assess than an assistant expected to answer every organisational question.