Automating the experiment check made our statistics invalid

Every company running experiments pays the same standing tax. Someone has to check on each live test, repeatedly: is it healthy, is the traffic split right, has anything moved, should we stop it. Then they have to translate the answer for PMs and leadership who reasonably do not want to read a confidence interval.

Multiply that by every running experiment, across several checkpoints each — day 1, day 3, day 7, day 14, day 21, day 30 — and it becomes a meaningful fraction of a data team's week. It is also work nobody enjoys, which means it gets done late, inconsistently, or not at all.

So we automated it. A daily job reads experiment configs, computes statistics, narrates them in plain language, and delivers the result to the team. No human in the loop.

The interesting part is not that we built it. It is what building it revealed: the manual process we were replacing had been quietly unsound the whole time, and automation is what made that impossible to ignore.

---

Two ways this goes wrong

Before any architecture, it's worth being precise about why naive automation here is actively harmful rather than merely unhelpful.

The statistical failure

An automated daily check evaluates every metric, on every experiment, at every checkpoint. That is textbook continuous peeking.

Under fixed-sample inference — a two-proportion z-test, Mann-Whitney, Welch confidence intervals — the nominal α of 0.05 is a guarantee about a single look at the data at a predetermined sample size. Recompute that test daily and each look is a fresh opportunity for a false positive. The errors compound. The true Type-I rate drifts far above the stated 5%, and nothing in the output signals that it has happened.

A previous generation of this pipeline did exactly this: fixed-sample frequentist tests recomputed daily across seven checkpoint windows. It produced results that looked rigorous. The p-values were real p-values. They just were not answering the question anyone thought they were answering.

This is worth sitting with, because the failure is not a bug. Every individual component was correct. The system was wrong because of how often it ran — which is precisely the property that made it valuable.

An automated system that peeks constantly is mathematically guaranteed to manufacture false positives, and it does so with the authority of automation. A human analyst producing a borderline result gets questioned. A dashboard producing the same result gets believed.

The communicative failure

The second failure is softer and just as damaging. Hand a PM a p-value and a power estimate and you will get confident misreadings in both directions:

• "p = 0.06, so it didn't work."
• "p = 0.04, so ship it."

Both are wrong. Both are the natural reading for someone who has not spent years internalizing what a p-value is and is not. A system that emits raw statistics at scale does not democratize rigor — it industrializes misinterpretation.

The architecture below is essentially two answers to these two problems, wired to a scheduler.

---

The shape: the warehouse is the runtime

The load-bearing architectural decision is that there is no application.

No service to deploy. No orchestration DAG. No model-serving layer. No application server.

• The transformation framework's scheduler (dbt Cloud-style, running daily) is the orchestrator.
• The warehouse is both the data plane and the LLM runtime. Modern warehouses expose an in-warehouse completion function — an AI_COMPLETE-style callable — usable from inside a model. The generation step is a SQL column.
• A single serverless function handles delivery.

Everything between config and delivery is a transformation DAG. That means dependency resolution, incremental builds, testing, lineage, and scheduling all come free from tooling the data team already operates.

For a small team this is a dramatic reduction in operational surface. There is no server that can be down. There is no deploy that can fail at 3am. The thing that runs your experiments analysis is the same thing that runs your nightly models, monitored the same way, debugged the same way, by the same people.

The tradeoff is real and worth stating plainly: you inherit the warehouse's latency and cost model, you cannot do anything the SQL dialect cannot express, and debugging a prompt means reading compiled SQL. For a daily batch workload that trade is clearly correct. For anything interactive it is clearly wrong — which is exactly why the conversational analytics agent I worked on has an entirely different shape.

The interactive/batch distinction is the single biggest determinant of agent architecture, and it is under-discussed. Most agent writing assumes an interactive loop. A large share of genuinely useful analytical automation is batch, and batch permits designs that interactive work cannot.

---

The pipeline

1 · Config lives where analysts already work. Each experiment is a page in a shared docs/wiki database: name, hypothesis, status, start date, primary and secondary metrics, control and treatment cohorts, and — importantly — which event proves a user actually saw the change, plus an optional free-form SQL predicate for cohort edge cases.

Two things this buys. Analysts configure experiments in a tool they already use, so there is no admin UI to build and maintain. And the config is reviewable by PMs without warehouse access.

One field — status — gates the entire pipeline. An experiment not marked in-progress simply does not exist downstream. That single field is the whole access control model, and it has never needed to be more complicated.

2 · Managed sync, not custom ETL. An off-the-shelf connector replicates the docs database into raw warehouse tables. A view pivots the tall property rows into one row per experiment, and that view is the declared source of truth for experiment configuration. No bespoke sync code exists anywhere in the system.

3 · Exposure and metric staging. Intermediate models assemble per-user, per-metric rows for exposed users across the product's event domains. The estimand is stated explicitly rather than left implicit — more on this below.

4 · The work queue. A model computes which experiments are due a check at which checkpoint today, producing per-user × per-metric rows. Being due is a property of data, not of a scheduler config. Adding a checkpoint is a change to one model, not a change to infrastructure.

5 · The stats engine. Entirely SQL plus warehouse UDFs. Emits one nested JSON object per experiment: metadata, an analysis block, and metrics × segments × cohorts.

6 · Generation. A model calls the in-warehouse LLM function with an assembled prompt, splitting one completion into a summary and a recommendation. A second call renders a self-contained HTML dashboard card.

7 · Publish gate. A mart table admits only non-empty results, stamped with the current date. Delivery inner-joins this table — so a partial or failed run delivers nothing rather than something stale. Fail-closed, enforced at the boundary where enforcement is cheapest.

8 · Delivery. A serverless function queries the gated results, uploads the HTML card to object storage, posts a chat message (header, summary, recommendation, a button to the full card), and appends a collapsible section to the experiment's page in the docs tool — idempotently, by reading existing blocks and skipping if that checkpoint is already written.

Idempotency from version one is not optional. You will re-run. Build it in before you need it, not after the day you double-post to forty experiment pages.

---

The statistics: why sequential testing is non-negotiable

This is the part worth understanding deeply, because it is where an automated experiment agent either earns trust or quietly destroys it.

The fix: anytime-valid confidence sequences

A single normal-mixture confidence-sequence family (Howard et al., 2021) covers all three metric shapes we needed:

A confidence sequence is valid at every look simultaneously. Daily peeking across many checkpoints becomes mathematically safe by construction rather than by convention.

This is the single change that makes the entire "automated daily check" concept legitimate. Everything else in the system is plumbing around it.

The cost is real: confidence sequences are wider than their fixed-sample equivalents at any given sample size. You pay for the right to look whenever you want. That is the correct trade for a system whose entire purpose is looking whenever you want, and it is a bad trade for a one-shot analysis. Knowing which situation you are in is the whole skill.

Dual reporting

Two p-values travel together, and being explicit about their different jobs prevents an entire category of argument:

• A sequential running-minimum p-value — the ship/stop decision anchor. A self-referencing incremental build reads the prior day's value and keeps the minimum, so evidence accumulates monotonically and cannot un-accumulate on a noisy day.
• A nominal fixed-sample p-value plus a chance-to-beat-control — the directional read people intuitively want.

One number decides. The other describes. Say which is which, in the schema, every time.

Sample-ratio mismatch as a soft-fail gate

A chi-square SRM test runs per experiment. Critically, it fails soft: metrics are still shown, with the caveat surfaced in the narration.

This is the same lesson that shows up everywhere in this kind of work. A gate that silently withholds results teaches people to distrust the tool and route around it. A gate that flags loudly and keeps going gets read.

Other deliberate choices

• Pooled-threshold winsorization, not trimming — trimming biases the interval.
• Bonferroni across secondary metrics within a segment.
• A minimum-sample gate below which results are marked insufficient rather than reported.
• Post-hoc power computed on the observed effect was removed entirely, as circular. Precision is communicated via interval width instead, which is the honest version of the same information.

That last one deserves emphasis. Observed-power is one of the most widely reported and least meaningful statistics in applied experimentation. Removing it is a small change that improves every downstream conversation.

Documented non-goals

The methodology spec enumerates what the engine deliberately does not do, each with its reason and its mitigation: variance reduction via regression adjustment (a power optimization, not a correctness requirement), FDR as an alternative to Bonferroni, multi-arm Dunnett correction, an intent-to-treat parallel track, and experiment-wide familywise error across primary plus secondaries.

A statistics spec that enumerates its own gaps is far more trustworthy than one implying completeness — and it stops the next engineer from assuming a guarantee that was never there. This practice is worth copying on its own merits, independent of anything else here.

Migration discipline

Replacing a live statistics engine used a three-step sequence:

1. Shadow mode. The new engine computes alongside the old. Nobody sees it.
2. Pre-registered A/A validation. Run it on experiments with no real effect and confirm the false-positive rate matches nominal.
3. Cutover that freezes methodology for in-flight experiments, so no test changes inference mid-flight.

Changing how significance is computed partway through an experiment invalidates that experiment. Any change to inference deserves this sequence.

---

Say what the number means

The pipeline commits explicitly to average treatment effect on the exposed — the effect among users who actually encountered the change, not everyone assigned.

Stating this matters more than which choice you make. "+6.5%" means something materially different under ATE-on-the-exposed than under intent-to-treat, and a PM reading a dashboard has no way to know which they are looking at unless the system tells them.

The known limitation is documented alongside it: conditioning on exposure is vulnerable when the treatment itself changes who gets exposed. Detecting that requires retaining assigned-but-unexposed users. That is flagged as a gap with a planned fix rather than papered over.

---

The prompt is data, not code

The templates live in a warehouse table. Two columns, a handful of rows: the narration template, the dashboard-card template, and the evaluation rubric. Nothing prompt-shaped exists in application code.

Editing a row deploys a new prompt. The next scheduled run picks it up. No release, no redeploy, no engineer.

This is genuinely double-edged, and I want to be honest about both edges. It collapses iteration time from hours to seconds and lets a non-engineer improve the output directly. It also means the most behavior-defining artifact in the system sits outside version control, with no review, no history, and no rollback.

The eval loop below is the control that makes this survivable. Without it, this design is reckless rather than clever. If you cannot build the eval loop, keep the prompt in the repo.

Assembly

Per experiment: a hard-coded schema-guidance prefix (teaching the model what the stats JSON's fields mean and how to phrase sequential results) + the narration template from the table + a conditional block keyed on experiment type + the serialized stats JSON. One completion, split on a delimiter into summary and recommendation columns. A second, separate call renders the HTML card.

Two structural notes worth stealing:

• Split the calls. Narrative and presentation are separate concerns with separate failure modes. One call doing both produces worse versions of each.
• The delimiter is the output contract. A single agreed marker splitting one completion into two typed columns is simpler and more robust than asking for JSON, when the shape is fixed and small. Reach for structured output when the shape is complex or variable — not reflexively.

---

The narration design

This is the part that generalizes furthest beyond experimentation.

Classify evidence into tiers before writing anything

Every key metric is sorted into one of:

• Strong evidence for
• Moderate — significant but one condition weak, and the prompt must flag which
• Weak — multiple conditions weak; explicitly unreliable, not a real signal
• No evidence of effect
• Counter-evidence of harm

Crucially, "not significant with adequate power" is framed as informative — evidence that no meaningful effect exists — rather than lumped in with "inconclusive." That distinction is where most experiment reporting goes wrong, and it is the difference between a null result that teaches the team something and a null result that gets quietly rerun until it turns positive.

Forcing classification before narration is what stops the model from writing a confident story around a noisy result. Only the top tier licenses a confident rollout recommendation.

Ban raw statistics from the output

No p-values. No power figures. No confidence intervals. No alpha thresholds.

Instead: uplift percentages, the underlying rates, sample sizes per group, and a plain-language reading of reliability.

This looks like dumbing down. It is the opposite. The audience is PMs and leadership; a p-value in that context is not information, it is a Rorschach test. The statistics are computed rigorously upstream, and the model's job is translation, not inference.

That framing is also why an LLM can be trusted with this task at all. It is never asked to decide significance. That decision is already made deterministically in SQL before the prompt is assembled. The model receives a JSON object where every hard call has been made, and turns it into prose.

The scope statement is explicit that humans make the final call. The system produces a recommendation, not a decision.

---

Evaluating a prompt that lives in a table

Because prompts deploy by UPDATE, quality control has to live somewhere else. It lives in the DAG:

• Regenerate each health check ~10 times with a fixed model.
• Score each run with a judge template — same table, different row — across several weighted dimensions.
• Aggregate into a composite score plus agreement rates on the recommendation and the direction across runs.
• Tag results with the git SHA and branch into an A/B results table, so prompt variants are comparable across time and a regression is attributable to a change.

Agreement is the measure that matters most. A prompt that produces a different recommendation on the same data across runs is disqualified regardless of how well any single run scores. Mean quality score will not catch that; run-to-run agreement will.

Two caveats an implementer should hold onto. An LLM judge inherits the biases of the model family judging it, so a judge sharing a lineage with the generator will over-reward familiar phrasing — treat the scores as relative signal between prompt variants, not as an absolute quality measure. And measure the thing people actually care about being stable, which is the recommendation, not the prose.

---

If you're building this

Ordered to de-risk the hardest parts first.

1. Config store. Create the experiment database with the property names your pivot expects — exact names matter, since the view pivots on them. For a prototype, skip the sync entirely and hand-insert rows shaped like the pivot's output.
2. Stats UDFs first. Implement the confidence-sequence functions as warehouse UDFs and validate them against known cases before anything else touches them. This is the correctness core; everything downstream is plumbing.
3. A/A validate. Run the engine against experiments with no real effect and confirm the false-positive rate matches nominal. Do this before generating a single narrative — if the statistics are wrong, better prose makes it worse.
4. Stats JSON. Assemble metrics × segments × cohorts into one object per experiment. Design this schema carefully; it is the interface between rigor and narration, and the prompt is written against it.
5. Prompt table and generation. Write the schema-guidance prefix first. Teaching the model what the JSON fields mean is most of the quality.
6. Publish gate, then delivery. Gate before you deliver, so a broken run ships nothing. Make the write-back idempotent from the first version.
7. Eval loop last, but actually build it. It is the only thing that makes an UPDATE-deployed prompt safe to change.

---

What this taught me

Let the model narrate; never let it infer. Every statistical decision — significance, tiering thresholds, gates — is computed deterministically upstream. This is why an LLM is appropriate here at all.

Automation forces better statistics than humans needed. A human checking an experiment weekly gets away with fixed-sample tests. An automated daily check does not. Automating an analytical process frequently exposes that the manual version was quietly unsound — and that discovery is often worth more than the automation.

Soft-fail gates get read; hard-fail gates get worked around.

Fail closed at the delivery boundary. A broken run should be silent, not misleading. Silence is a fine failure mode. Confident staleness is not.

Banning raw statistics improved decisions. Precision in the output is not the same as usefulness.

Prompt-as-data is powerful and dangerous in equal measure. Only pair it with an automated eval loop.

Measure agreement, not just quality.

Enumerate your non-goals.

Migrate inference methods in shadow mode.

Config belongs where the humans are. Experiment definitions in the tool analysts and PMs already use, synced by a managed connector, meant zero admin UI and zero bespoke sync code. The integration cost is real; building and maintaining a config app is much larger.

---

This was the first agent we shipped, and it remains the one I point to when someone asks whether autonomous analysis is safe. It is — under a specific and narrow condition: that the model is never the thing making the call. The next system I built could not meet that condition, and needed an entirely different set of defenses.