Deploying our first natural-language analytics agent

Analytics teams get interrupted constantly with questions that are individually cheap and collectively enormous. What's DAU on that platform this week. Did the checkout change move conversion. Why did retention dip on Tuesday. Each takes an analyst ten minutes and destroys an hour of focus.

The obvious fix, pointing an AI model at Snowflake, fails in a specific and instructive way. The model writes plausible SQL against tables it does not understand. It combines data at the wrong level of detail. It adds up daily snapshots that were never meant to be added. And it returns a confident, beautifully formatted, wrong number.

A wrong answer delivered fluently is worse than no answer, because it gets pasted into a deck.

So the problem is not “can a model query Snowflake.” It is: how do you make a model's analytical output trustworthy enough that a stranger can act on it? Everything distinctive about this system follows from that one question.

The constraints that shaped it

Every number needs a source. Each figure must trace back to a query the agent actually ran. When it can't back something up, it says so instead of guessing.

The rules that make an answer correct live in analysts' heads. This metric is weekly, not daily. That platform's spend data lags three days. This column mixes two kinds of customer. There is far more of this than the model can read at once, so it has to read selectively.

Real questions take minutes; Slack expects a reply in seconds. That mismatch shapes how every request flows through the system.

The interface is a Slack thread, not a dashboard. Follow-up questions come for free, but only if the system remembers what the thread has already established.

The team is small. Every operational choice favored “one person can debug this on a Friday” over sophistication.

From mention to answer

A question starts as an @mention in Slack. Because a real investigation takes longer than Slack will wait, the agent replies twice: an instant acknowledgment, then the answer in the same thread a few minutes later. In between, it gathers context, queries the data, and checks its own answer. Step through it below, or click any component:

The fast path matters more than it looks. A handful of common questions make up a large share of traffic, and they're answered in under three seconds without ever reaching the full agent. Fast replies on common questions do more for adoption than better answers on hard ones.

Where the agent runs

The agent itself is one long-running program on its own server, because an investigation outlasts the time limit of the lightweight function that receives Slack messages.

Two tiers of model. Most questions go to a fast, inexpensive model. Harder ones go to a deeper model that costs about ten times more and takes about ten times longer. When the system is unsure which a question needs, it escalates, because overpaying for an easy question is far cheaper than a weak answer to a hard one.

Hard limits on every run. Each run has caps on time, steps and cost. The limit people forget is a timeout between messages: a stuck run doesn't raise an error, it just goes quiet, and without that check it would sit there until the outermost limit.

Visible progress. While it works, the agent keeps updating its “working on it” message with what it's doing. Three minutes of silence reads as broken; the same three minutes with visible progress reads as thorough.

The knowledge layer

This is what separates a demo from a working system, and it's the part most imitations skip. The tempting approach is to hand the model every rule, table description and caveat up front. That stops fitting quickly, and it also makes answers worse, because the model latches onto rules that have nothing to do with the question.

More context makes the model worse. An irrelevant rule in context is not neutral.

Plain-text playbooks

Analysts write down what they know in ordinary text files, organized by topic, and add short markers for the parts the agent should index:

• RULE: a correctness constraint (“weekly retention means week one, not day seven”)
• ROUTE: which table answers which kind of question
• TOPIC: what a section is about, for matching

Because each marker sits right next to the explanation it belongs to, analysts don't have to learn a special format, and the two can't drift apart.

Choosing what's relevant

When the agent starts, it indexes every playbook by topic, along with the metric definitions from dbt. For each question, two matchers pick the relevant sections. A small model handles paraphrase: “why are people churning” means retention. Plain keyword matching catches internal jargon and product names the model has never seen. They fail in different ways, so running both is cheap and clearly better than either alone. The result is trimmed to a fixed budget before the agent sees it.

An always-included minimum

A tiny set of facts goes along with every question regardless of topic, because getting them wrong would be unrecoverable. The discipline is keeping that set genuinely tiny.

Read before you query

The agent can't run a single query until it has read at least one playbook. Without that rule, models skip the reading and improvise, exactly like people. Sometimes the fix for “the model doesn't reliably do X” is making X a requirement rather than an instruction.

Knowledge that updates itself

Some knowledge changes too often to maintain by hand, like the list of dashboards and the tables behind them. It's rebuilt automatically on a schedule, in the same format as the playbooks. The danger is that it goes stale silently: a login expires, the rebuild quietly stops, and answers drift from reality with no error anywhere. Alert on staleness, not only on failure. “Last successful rebuild was 12 days ago” catches this. “Did the last run error” does not.

A timeline of what happened

A reviewed timeline of launches, experiments and incidents lets the agent connect “conversion dropped on the 12th” to “the checkout change shipped on the 11th.” That's the difference between reporting a metric and explaining it. Two rules keep it safe:

• A person approves every new entry. A bad entry wouldn't just be wrong once; it would quietly shape every future answer.
• The agent cites entries by ID, and the system fills in the links. A model that never handles a URL can never invent one.

Memory for each thread

Each Slack thread keeps notes on what's been established: the tables used, queries that worked, key numbers. That's what lets “now break that out by platform” work without starting over.

Checking the answer

Before anyone sees an answer, it goes through a fixed series of checks. There's deliberately no second AI model involved: a checker that can make things up is worse than none, and this step has to be easy to debug at 9am when someone says the bot lied.

Clean up

Strip out leaked reasoning (“let me check...”) and unrequested offers to help further, move tables out of the prose into proper tables, and rebuild a missing table from the raw query results.

Rebuild the sources

The list of sources is rebuilt from the log of queries that actually ran, not from what the model says it did, and the real date range is read from the query itself. Never let the model report its own sources. Models are honest in general and unreliable in the specifics, and sources are all specifics.

Flag what isn't supported

Every number is checked against queries that actually succeeded. Anything unsupported lowers the answer's confidence and gets a visible caveat.

The gate appends a caveat. It never deletes an answer.

An earlier, stricter version replaced doubtful answers with “I could not verify this,” and it kept destroying correct answers, because no detector is perfect. A caveat is recoverable: the reader still gets the analysis and knows to double-check it. A deleted answer is not. When your detector is imperfect, choose the mechanism whose mistakes are survivable.

Confidence and retry

A confidence score combines the warning signs, such as errors, sources that disagree, or the agent repeating itself, and low scores carry a caveat. Some failures trigger a retry instead of shipping at all, like an answer to a data question that contains no data.

Turn known mistakes into code

Some mistakes recur forever. The classic: adding up a week of daily snapshots of a running total, which inflates the number several times over. Every new analyst makes it, and so does every model. A written rule helps; a check that rejects that query outright is what actually stops it. Knowledge that can be written as code should be code.

Staying right over time

Every answer has a thumbs up and down, and a thumbs-down asks what was wrong. Those corrections are the most valuable signal the system produces: a domain expert explaining exactly what the agent got wrong is worth more than a thousand thumbs-ups. Each one becomes a proposal for a person to review: a fix to the playbooks, plus a new test question so the same mistake can't quietly return.

A half-connected learning loop is not a learning loop. Capturing corrections is the easy part. Decide up front who reviews the proposals and when, or they pile up where nobody reads them.

Running it safely

Scheduled work uses the front door. Daily digests, weekly scorecards and alert triage go through the same path as human questions, so there's one pipeline to trust and debug. As a bonus, they exercise that path all day, so a problem shows up on a schedule instead of in front of a user.

Real questions before every release. Each release runs a set of real questions against a full test copy and checks the answers: sources present, no leaked reasoning, a caveat when support is missing, acceptable speed. Ordinary code tests can't tell you the agent quietly stopped citing sources or got 40% slower. This can.

A test copy anyone can reach. Adding a short prefix to a Slack message sends that one question to the test copy instead. Real interface, real data, zero risk to the live system.

Releases wait their turn. A release holds until no questions are in progress, so shipping never cuts off someone's three-minute investigation.

If you're building this

Build in this order. It front-loads the parts that decide whether the thing works at all.

1. 01Prove the loop. One data source, one hardcoded question, a command-line tool.
2. 02Make it wrong loudly. Record which queries backed which claims before you have users.
3. 03The knowledge layer. Where answer quality jumps.
4. 04Chat, with two replies. The instant acknowledgment, the answer later, and progress in between.
5. 05The checks. Clean up, rebuild sources, score confidence. Resist making them delete answers.
6. 06Tests, then everything else. A dozen real questions that must pass before each release.

What I'd do differently

Pick one scheduler on day one. Four different ways of scheduling jobs piled up, each the easiest choice at the time. Together they meant no single place showed what runs when, and a job set up on a server but not saved in the code could vanish without a trace.

Treat documentation as code that rots. At one point the agent's own instructions pointed it at a tool that had been switched off, and the model followed them faithfully off a cliff. If a document names a tool, a setting or a server, add a test that checks it still exists. Documentation that can't fail will eventually lie.

Finish migrations. For a while, two versions of the agent's core loop lived side by side, with a setting deciding which one was live. Migrations should end in a deletion. Until they do, they are not finished.

Lock things down early. Passwords belong in a proper secrets store, and the agent's server belongs behind encryption. Both take an afternoon, and both are easy to skip when there's only one server.

The experiment agent that came before this one could run without oversight because the model never made a call: every decision was computed in SQL inside Snowflake first. This system can't make that guarantee, and everything above is what replaces it.