Guide · AI in Institutions
How to evaluate an internal knowledge assistant before release
An internal AI assistant is only as good as its documents and its tests. Evaluate it against real questions, including the ones it must refuse, before anyone relies on it.
The short answer
Before release, test a knowledge assistant against a set of real questions with expected answers and sources, plus questions the documents cannot or must not answer. It passes when its answers are correct, cite the right document, and decline what they should. Fix content ownership and permissions first; they cause many of the failures.
Readiness comes before the model
Many assistant failures are really content failures: outdated documents, contradictory versions and material that should never have been indexed. Give every document an owner and a review date, remove duplicates, and map who may see what.
Build the evaluation set
Collect real questions from inboxes, help desks and staff. For each, record the document that answers it and the expected answer. Add questions the assistant must refuse: personal data, confidential material, anything outside the documents.
What to measure
- Correctness: is the answer right?
- Citation: does it point to the document that supports it?
- Refusal: does it decline what it should, rather than guess?
- Leakage: can anyone extract material they should not see?
A worked example
The AI knowledge assistant readiness checklist scores an association planning a member help assistant. It knows the questions members ask, but it has no test set with expected answers, and its starter evaluation set holds a single refusal question. Until a proper test set exists, there is no way to know whether the assistant is safe to launch.
What this guide does not cover
Passing a test set does not guarantee future answers. Monitor after launch and re-test when documents change.
Responsible for this guide
Examples in this guide are illustrative, not client results. Figures come from the free tool’s worked example; change the inputs in the tool to see your own.
Related service
The same method, delivered with you: scoped in writing, with a named principal and a fee agreed before any work begins.
More from AI in Institutions
What needs to move forward?
Tell us the decision, the challenge or the opportunity. We reply with a scoped approach, a named principal and a fee before any work begins.
No charge to submit an enquiry.