How to evaluate an AI support tool for your protocol: a buyer's checklist

Every AI support tool demos well. The vendor pastes in a clean question, the model returns a fluent answer, and it looks like the support problem is solved. Then it ships, a real user asks about their own failed transaction, and the tool confidently states a balance that is not theirs. The gap between the demo and production is where protocol teams lose money and trust, and it is almost always invisible in a sales call.

Why the usual criteria mislead

Buyers tend to evaluate AI support on three things: which model it uses, how good the writing is, and how fast it responds. All three are easy to demo and none of them predict the failure that actually hurts you.

The damaging failure in support is not a clumsy sentence. It is a confident, specific, wrong claim about a user's own situation: their balance, their transaction status, whether their funds are safe. A better model writes that wrong claim more fluently. Faster responses deliver it sooner. The variable that matters is whether the tool checked the user's actual situation before it answered, and that is precisely the thing a scripted demo hides. (For the underlying distinction, see our piece on what agentic support actually is.)

The five-part checklist

1. Can it read the user's actual situation, or only your docs?

This is the first fork and it eliminates most tools. A retrieval chatbot can only answer from documentation and past tickets. It cannot see this user's transaction, this wallet's balance, or the contract's current state. Ask directly: when a user says 'my swap failed', does the tool fetch that transaction and inspect it, or does it summarise generic text about swaps failing? If it cannot read live state, every answer about a specific user is a guess dressed as a fact.

• Green flag: it fetches the transaction, replays it, and describes what happened to that user.
• Red flag: every answer is a paraphrase of your documentation, however well written.

2. Can it show the evidence behind a specific answer?

In the demo, ask a question and then ask: what did you check to produce that answer, and can you show me? A tool built to investigate can point at the transaction it read, the contract state it replayed against, and the documentation it drew on. A tool built to generate cannot, because there was nothing behind the answer except the model's training. This one question is the fastest way to tell the two apart.

If a vendor cannot show you what their tool checked before it answered, they are selling you fluency, not support.

3. What does it do when it does not know?

Ask the tool something genuinely unanswerable from any system it can read, or something outside its scope. Watch what it does. The correct behaviours are to decline clearly, or to escalate to a human with the context attached. The dangerous behaviour is to improvise a plausible answer anyway. A support tool that never says 'I do not know' is not confident, it is unaware of its own limits, and it will invent an answer about your users' funds with the same fluency it invents everything else.

4. What does it record, and could an auditor replay it?

Any tool that answers questions about money creates an obligation: to establish, later, what it saw when it answered. This matters for two reasons. Internally, it is how you improve the tool and catch bad answers. Externally, it is what you show a partner, an auditor, or a regulator who asks 'what did your support system tell users during that incident?'. Ask what is recorded per answer.

• Which checks ran, and which failed: a lookup that errored is not the same as one that returned nothing.
• The state the answer was true as of, so it can be reproduced, not just re-generated.
• Whether the answer was grounded in a real read at all, computed from what ran rather than the model rating its own confidence.
• An integrity guarantee: can a stored answer be quietly edited after the fact, or is the record tamper-evident?

Press on the wording here. 'Verified' and 'sourced' are not the same claim. Sourced means a figure came from a real read. Verified means every claim is entailed by the evidence, which very few systems can honestly assert today. A vendor who says 'verified' when they mean 'sourced' is overclaiming on exactly the axis you are buying for.

5. Is it read-only, and what can it be tricked into doing?

A tool that can act on live systems is a tool that can be manipulated into acting. For DeFi support specifically, confirm it is read-only by default: it should read chain state and answer, never hold keys or move funds on a user's say-so. If the tool can take on-chain actions, ask how those are gated, and treat both user input and the on-chain data it reads as untrusted, because a hostile token name or transaction memo is an injection vector. Support that can move money is a different, larger risk purchase than support that can only explain.

Questions that expose a weak tool in the demo

Bring your own examples to any demo. Vendors curate the questions that work; you should ask the ones that break:

• Paste a real failed transaction hash from your own protocol and ask why it failed. A generic answer means it did not actually read it.
• Ask about a claim window or vesting schedule specific to your contracts. If it answers from general knowledge instead of your data, retrieval is shallow.
• Ask it something false as if it were true ('why did my withdrawal of 500 tokens fail?' when no such transaction exists). A good tool checks and corrects you; a weak one accepts the premise and invents a reason.
• Ask for financial advice ('should I sell?'). It must refuse, every time, including when the question is reframed as a hypothetical.

On pricing: seats vs resolutions

How a tool is priced tells you what era it is from. Seat-based pricing is a holdover from software a human operates: you pay per agent using the dashboard. A tool that resolves questions itself has a more honest unit, the resolution: you pay for questions that actually got answered, and cost scales with your users' activity rather than your headcount. Neither is wrong, but seat pricing on a tool that claims to be autonomous is worth a raised eyebrow.

The one-line test

If you remember nothing else: ask the vendor to answer one hard question about a real transaction, then ask them to prove it. The tools worth buying will show you the transaction they read, the state they checked, and the sources behind the answer. Everything else is a language model with a support-shaped prompt, and you will discover the difference in production, on your users, at the worst possible time.