Start with the questions, not the tool
Write down who will ask and what they ask. A developer stuck on an authentication error, a customer asking whether a plan includes an export, and an engineer asking why a service retries three times are three different jobs. They need different sources, different places to ask and different tolerance for a wrong answer.
Collect thirty real questions from your support inbox, community forum and internal chat. Include ten you know your documentation answers well, ten that need information from two places, and ten it does not answer at all. This set is the whole evaluation. Everything below is about how to use it.
Check what it can read, and how fresh it stays
An assistant is only as good as the content behind it. List where your knowledge lives: the docs site, code repositories, API schemas, PDFs, a help centre, a forum. Then check each is supported without a workaround.
- Connection. Does it read the source directly, or do you upload copies that will go stale?
- Freshness. How often is each source re-read, and what happens to a page you delete?
- Access. Public sources are easy. Private ones need a credential, so ask where it is stored and who can see it.
- Scope. Can you keep one set of sources for customers and another for staff, so an internal runbook never appears in a public answer?
Judge answers by their evidence
Put your thirty questions to each tool, connected to the same content. For every answer, ask two things: is it correct, and can you prove it from what it shows you?
The second matters more than it sounds. An answer you must verify by searching the docs yourself saves nobody time. Look for citations that point to the passage that supports the claim, not just to a page that mentions the topic. Click several. A citation that does not support the sentence beside it is a warning sign, because it means the answer came from somewhere else.
Then look at the ten questions your documentation does not answer. A good assistant declines, and says so plainly. A weaker one writes something plausible. This single test separates tools faster than anything else, and it is the one vendors are least likely to show you. The companion post on citing sources and abstaining explains why.
Count only the places you will use
Vendors list integrations. You need three or four: perhaps a widget on your docs site, an API so your product can answer in-app, and an editor integration for developers. If your developers use AI coding assistants, ask whether the tool offers an MCP server, and whether it is read-only. Read-only is the safe default for a tool that only needs to search.
Check the cost of each place, too. Some plans include the widget and charge extra for the API.
Ask the security questions early
Security reviews end more purchases than missing features do, so start them in week one.
- Tenant separation. How is one customer’s content kept apart from another’s? Ask what happens when a request carries no organisation.
- Where data lives. Which country, and can you pin it? Can you run the platform yourself if you must?
- Secrets. What stops an API key committed to a repository from being indexed and quoted back in an answer?
- Hostile content. Your sources can include text written by outsiders. Ask how instructions hidden in a page are kept from steering the assistant.
- Certifications. If you need one, ask which the vendor holds today, not which it is working toward. A vendor with none should say so and answer your questionnaire directly.
Work out the cost at your volume
Pricing comes in two shapes. A flat plan with an answer allowance is easy to budget. Usage-based pricing can be cheaper at low volume and surprising at high volume. Estimate your monthly questions from your support volume and traffic, then ask each vendor for a quote at that number and at three times that number.
Also ask what happens at the limit: does the assistant stop answering, slow down, or bill overage? And what is included in the price, such as analytics, extra sources and extra seats.
Run a two-week pilot, then read the analytics
Put the shortlisted tool in front of a small group with real questions. The most useful output is not the answers but the list of questions the assistant declined. Each one is a gap in your documentation, ranked by how often people ask. Check that the tool shows you this, and shows it without naming who asked.
MITHUNAI RAG is built around these checks: every answer cites its sources, it declines when the content has no answer, and its source analytics and unanswered-questions view show where your docs fall short. How we compare with other tools is on the alternatives page, and the pricing page lists every plan. To run your thirty questions on your own content, book a demo.