The experiment agent that never takes a day off

Every company running experiments pays the same standing tax. Someone has to check on each live test, repeatedly: is it healthy, is the traffic split right, has anything moved, should we stop it. Then they have to translate the answer for PMs and leadership, who reasonably do not want to read a confidence interval.

Multiply that by every running experiment, across several checkpoints each (day 1, 3, 7, 14, 21, 30), and it becomes a meaningful fraction of a data team's week. Nobody enjoys the work, so it gets done late, inconsistently, or not at all.

So we automated it. A statistical engine reads experiment configs, applies a sequential testing methodology, explains the results in plain language, and delivers them to the team. No human in the loop, and analysts got back roughly half the time they had spent on health checks.

Building it revealed something more important: the manual process we were replacing had been unsound the whole time, and automation made that impossible to ignore.

Two ways this goes wrong

Naive automation here is actively harmful, not just unhelpful, for two reasons.

The statistical failure

An automated daily check evaluates every metric, on every experiment, at every checkpoint. That is textbook continuous peeking.

Under fixed-sample inference (a two-proportion z-test, Mann-Whitney, Welch confidence intervals), the 5% false-positive rate is a guarantee about a single look at the data at a predetermined sample size. Recompute that test daily and each look is a fresh chance for a false positive. The errors compound, and nothing in the output shows it.

A previous version of this pipeline did exactly this: fixed-sample tests recomputed daily across seven checkpoint windows. The results looked rigorous, and the p-values were real p-values. They just weren't answering the question anyone thought they were.

The failure wasn't a bug. Every component was correct. The system was wrong because of how often it ran, which is exactly what made it valuable.

An automated system that peeks constantly will manufacture false positives, and it does so with the authority of automation. A human analyst with a borderline result gets questioned. A dashboard with the same result gets believed.

The misread p-value

Hand a PM a p-value and a power estimate and you get confident misreadings in both directions:

• "p = 0.06, so it didn't work."
• "p = 0.04, so ship it."

Both are wrong, and both are the natural reading for someone who hasn't spent years with statistics. A system that sends raw statistics to everyone doesn't spread rigor. It spreads misinterpretation.

The architecture below answers these two problems, wired to a scheduler.

Snowflake is the runtime

The key architectural decision is that there is no application. No service to deploy, no orchestration layer, no model-serving layer, no application server.

• The dbt Cloud scheduler, running daily, is the orchestrator.
• Snowflake is both the data plane and the LLM runtime. Its in-warehouse completion function, Cortex AI_COMPLETE, can be called from inside a model. The generation step is a SQL column.
• A single serverless function handles delivery.

Everything between config and delivery is a transformation DAG, so dependency resolution, incremental builds, testing, lineage, and scheduling come from tooling the data team already runs. There is no server that can be down and no deploy that can fail at 3am.

The tradeoff: you inherit Snowflake's latency and cost model, you can't do anything SQL can't express, and debugging a prompt means reading compiled SQL. For a daily batch job that's the right trade. For anything interactive it's wrong, which is why the analytics agent has a completely different shape.

Whether a job is interactive or batch is the biggest factor in agent architecture. Most writing on agents assumes an interactive loop, but a lot of useful analytical automation is batch, and batch allows simpler designs.

Step through the daily run below, or click any component:

The pipeline

1. 01Config lives where analysts already work. Each experiment is a page in a shared Notion database: hypothesis, status, start date, metrics, cohorts, and which event proves a user actually saw the change. No admin UI to build, and PMs can review config without Snowflake access. The status field gates the whole pipeline. It is the entire access-control model, and it has never needed to be more complicated.
2. 02Managed sync, not custom ETL. Fivetran replicates Notion into Snowflake. A view pivots it into one row per experiment, and that view is the source of truth for configuration.
3. 03Exposure and metric staging. Models assemble per-user, per-metric rows for exposed users across the product's event domains.
4. 04The work queue. A model computes which experiments are due a check today. Being due is a property of the data, so adding a checkpoint is a change to one model, not to infrastructure.
5. 05The stats engine. SQL plus Snowflake UDFs. Emits one nested JSON object per experiment: metadata, analysis, and metrics × segments × cohorts.
6. 06Generation. Cortex turns the stats into a summary and a recommendation. A second call renders an HTML dashboard card.
7. 07Publish gate. Only non-empty results stamped with today's date get through, so a partial or failed run delivers nothing rather than something stale.
8. 08Delivery. A serverless function posts to Slack and appends the result to the experiment's Notion page, skipping any checkpoint already written. Build that idempotency in from version one. You will re-run, and you don't want to double-post to forty experiment pages.

Why sequential testing is superior to fixed-horizon testing

This is where an automated experiment agent earns trust or loses it.

A single family of anytime-valid confidence sequences (Howard et al., 2021) covers all three metric shapes we needed:

A confidence sequence is valid at every look at once, so daily checking is safe by construction. This one change is what makes an automated daily check legitimate. Everything else is plumbing around it.

The cost: confidence sequences are wider than fixed-sample intervals at the same sample size. You pay for the right to look whenever you want. That's the right trade for a system built to look every day, and the wrong one for a one-off analysis.

Two p-values, two jobs

• A sequential running-minimum p-value anchors the ship/stop decision. Each day's build keeps the minimum of the prior value, so evidence can't un-accumulate on a noisy day.
• A nominal fixed-sample p-value plus a chance-to-beat-control gives the directional read people want.

One number decides, the other describes. Say which is which, in the schema, every time.

Sample-ratio mismatch fails soft

A chi-square SRM test runs per experiment. If it fails, metrics are still shown, with the caveat in the write-up. A gate that silently withholds results teaches people to route around the tool. A gate that flags loudly and keeps going gets read.

Other deliberate choices

• Pooled-threshold winsorization, not trimming, because trimming biases the interval.
• Bonferroni across secondary metrics within a segment.
• A minimum-sample gate below which results are marked insufficient.
• Post-hoc power was removed entirely, as circular. Interval width communicates precision instead. Observed power is one of the most reported and least meaningful statistics in applied experimentation.

Documented non-goals

The methodology spec lists what the engine deliberately does not do, each with a reason and a mitigation: regression-adjusted variance reduction, FDR as an alternative to Bonferroni, multi-arm Dunnett correction, an intent-to-treat parallel track, and experiment-wide familywise error. A spec that lists its own gaps is more trustworthy than one that implies completeness, and it stops the next engineer from assuming a guarantee that was never there.

Migrating a live stats engine

1. Shadow mode. The new engine runs alongside the old. Nobody sees it.
2. Pre-registered A/A validation. Run it on experiments with no real effect and confirm the false-positive rate matches 5%.
3. Cutover that freezes methodology for in-flight experiments, so no test changes inference partway through.

Say what the number means

The pipeline commits to the average treatment effect on the exposed: the effect among users who actually saw the change, not everyone assigned. Stating it matters more than which choice you make. "+6.5%" means something different under intent-to-treat, and a PM can't tell which unless the system says so.

The known limitation is documented alongside it. Conditioning on exposure breaks down when the treatment changes who gets exposed, and detecting that requires keeping assigned-but-unexposed users. It's flagged as a gap with a planned fix.

The prompt is data, not code

The templates live in a Snowflake table: the write-up template, the dashboard-card template, and the evaluation rubric. Editing a row deploys a new prompt. The next scheduled run picks it up. No release, no redeploy, no engineer.

This cuts both ways. Iteration drops from hours to seconds and a non-engineer can improve the output directly. But the artifact that most defines the system's behavior sits outside version control, with no review, history, or rollback.

The eval loop is what makes this safe. If you can't build the eval loop, keep the prompt in the repo.

Each experiment's prompt is assembled from a fixed schema guide (what the stats JSON fields mean), the template from the table, a block for the experiment type, and the stats JSON itself. Two notes on structure:

• Split the calls. The write-up and the dashboard card fail in different ways. One call doing both produces worse versions of each.
• A delimiter is the output contract. One agreed marker splitting a completion into two columns is simpler than asking for JSON when the shape is small and fixed.

Classify first, then write

This part applies well beyond experimentation. Before writing anything, every key metric is sorted into a tier:

Treating a well-powered null as informative, not "inconclusive", is where most experiment reporting goes wrong. It's the difference between a null result that teaches the team something and one that gets rerun until it turns positive. Classifying before writing stops the model from telling a confident story about a noisy result.

No raw statistics in the output

No p-values, power figures, confidence intervals, or alpha thresholds. Instead: uplift percentages, the underlying rates, sample sizes per group, and a plain-language read on reliability. For PMs and leadership, a p-value invites misreading. The statistics are computed rigorously upstream, and the model's job is translation, not inference.

That is also why an LLM can be trusted here. It never decides significance. That decision is made in SQL before the prompt is assembled, and humans make the final call. The system produces a recommendation, not a decision.

Evaluating a prompt that lives in a table

Because prompts deploy with an UPDATE, quality control lives in the DAG:

• Regenerate each health check about 10 times with a fixed model.
• Score each run with a judge template (same table, different row) across several weighted dimensions.
• Aggregate into a composite score plus agreement rates on the recommendation and direction across runs.
• Tag results with the git SHA and branch, so prompt variants are comparable and a regression can be traced to a change.

Agreement matters most. A prompt that gives different recommendations on the same data across runs is disqualified, however well any single run scores. Treat judge scores as a comparison between prompt variants, not an absolute measure, since an LLM judge favors phrasing from its own model family.

If you're building this

Build in this order. It de-risks the hardest parts first.

1. 01Config store. Property names must match what the pivot expects. For a prototype, hand-insert rows and skip the sync.
2. 02Stats UDFs. Validate the confidence-sequence functions against known cases before anything else touches them.
3. 03A/A validate. Confirm the false-positive rate is 5% before generating a single write-up. If the statistics are wrong, better prose makes it worse.
4. 04Stats JSON. Design this schema carefully. It's the interface between the statistics and the write-up.
5. 05Prompt table and generation. Write the schema guide first. Teaching the model what the fields mean is most of the quality.
6. 06Publish gate, then delivery. Idempotent write-back from the first version.
7. 07Eval loop last, but build it. It's the only thing that makes changing a table-deployed prompt safe.

What this taught me

Let the model explain; never let it infer. Significance, tier thresholds, and gates are all computed deterministically upstream. That's why an LLM belongs here at all.

Automation demands better statistics than humans did. A weekly human check gets away with fixed-sample tests. A daily automated one doesn't. Automating an analytical process often shows the manual version was unsound, and that discovery can be worth more than the automation.

Soft-fail gates get read. Hard-fail gates get worked around.

Fail closed at delivery. A broken run should be silent, not misleading.

Removing raw statistics improved decisions. Precision in the output isn't the same as usefulness.

Prompt-as-data is powerful and risky. Only pair it with an automated eval loop. Measure agreement, not just quality. List your non-goals. Migrate inference methods in shadow mode.

Config belongs where the people are. Experiment definitions in the tool analysts and PMs already use meant no admin UI and no custom sync code.

This was the first agent we shipped, and it's the one I point to when someone asks whether autonomous analysis is safe. It is, under one narrow condition: the model is never the one making the call.