How to build a chat-native agent that answers business questions by planning and executing real queries against a data stack — and, more importantly, the machinery that keeps it from confidently returning a wrong number. Hover any component or wire for what it does and why it exists. Click to pin. Use the flow buttons to trace one path at a time.
The choices that separate a system people trust from a demo that impresses once. Each is a place where the obvious approach is wrong.
An earlier evidence gate replaced unsupported-looking answers outright — and repeatedly destroyed correct ones, because detecting "is this claim supported" has imperfect recall. Lowering confidence and attaching a caveat is recoverable; deleting a right answer is not. When your detector is imperfect, choose the mechanism whose false positives are survivable.
A model checking a model adds a second thing that can hallucinate and can't explain itself. Cross-referencing claims against a tool-execution log is boring code that's right every time — and debuggable at 9am when someone says the bot lied.
An irrelevant rule in context isn't neutral — the model pattern-matches on it and drags an unrelated table into the answer. Select aggressively by topic, then trim to a line budget. The gap between "selected" and "everything" widens as the corpus grows.
Build citations from the tool log; parse the data window out of the executed SQL. Sharpest version: the agent cites context by id and the system resolves ids to links — a model that never handles a URL can never invent one.
Domain experts won't maintain a schema. Plain-text playbooks with inline RULE/ROUTE/TOPIC markers let analysts write explanation while the machine indexes the marked parts — and keeps the rule next to the reasoning, so they can't quietly diverge.
Chat wants an HTTP response in ~3 seconds; real analysis takes minutes. No synchronous design satisfies both. Once you accept that the acknowledgment and the answer travel different paths, the rest follows: expected read timeouts, out-of-band delivery, a visible placeholder.
No test suite tells you the agent quietly stopped citing sources. Only real questions run against a live staging instance with real tool connections, asserting a shared quality contract, wired as a deploy gate. Slow, occasionally flaky, irreplaceable.
Some mistakes are structural and forever — summing daily snapshot rows of a cumulative metric multiplies the truth. Every new analyst makes it; so does every model. A playbook rule helps; a guard in the query path that rejects the shape is what stops it.
A generated index whose credential expired fails quietly, and the agent keeps answering from a frozen snapshot for weeks. "Last successful build was N days ago" catches it; "did the last run error" does not.
Three minutes of silence reads as broken. The same three minutes with a placeholder that names the tool currently running reads as thorough. Cosmetic, cheap, and it changes how people describe the tool.
Dispatch scheduled prompts through the same endpoint humans hit. One pipeline, one set of guarantees — and scheduled jobs exercise the user path continuously, so regressions surface on a schedule instead of on a person.
A domain expert writing "wrong table — use X" is the most valuable signal the system produces. Capture it, turn it into a proposal plus an eval case — but decide up front who reviews proposals and when. Otherwise corrections accumulate in a directory nobody opens.