A daily agent that reads A/B test configs, computes anytime-valid statistics, narrates them for a non-technical audience, and delivers the result — running entirely as a warehouse transformation DAG plus one delivery function. Hover any component or wire for what it does and why. Click to pin; use the flow buttons to trace one path.
The decisions that matter, and the reasoning behind each. Several invert what you'd reach for by default.
A human checking an experiment weekly gets away with fixed-sample tests. A system that re-evaluates every metric every day is continuous peeking — Type-I error inflates far above nominal, silently, with the authority of automation. Anytime-valid confidence sequences are what make the concept legitimate at all.
Every statistical decision — significance, evidence tier, gates — is computed deterministically in SQL before the prompt is assembled. The model receives a JSON object where the hard calls are already made and translates it into prose. That's precisely why an LLM is appropriate here.
No p-values, no power, no confidence intervals. Instead: uplift, the underlying rates, sample sizes, and plain-language reliability. The audience is PMs and leadership, where a p-value isn't information — it's a Rorschach test. Precision in output is not the same as usefulness.
Each metric is sorted into strong / moderate / weak / no-effect / harm before any prose is generated, and only the top tier licenses a confident rollout call. "Not significant with adequate power" is framed as informative — evidence of no meaningful effect — not lumped in with "inconclusive."
Templates in a warehouse table mean editing a row deploys a new prompt — seconds instead of hours, and a non-engineer can improve the output. It also puts the most behavior-defining artifact in the system outside version control, with no review or rollback. Only do this if you also build the eval loop.
The sample-ratio-mismatch check flags loudly and shows the metrics anyway rather than withholding them. A gate that silently blocks results teaches people to route around the tool; one that caveats loudly gets taken seriously.
A publish gate admits only complete results stamped with the current date, and delivery inner-joins it. A broken run therefore ships nothing. Silence is an acceptable failure mode; confident staleness is not.
The eval loop regenerates each check ~10× and checks whether the recommendation and direction hold across runs. A prompt that flips its recommendation on identical data is disqualified regardless of how well any single run scores — instability that a mean quality score misses entirely.
The estimand is declared explicitly — effect among users who actually saw the change, not everyone assigned. "+6.5%" means different things under different estimands, and a reader can't tell which unless the system says so.
The methodology spec names what it deliberately doesn't do — variance reduction, FDR, multi-arm correction, intent-to-treat — each with its reason and mitigation. A spec that admits its gaps is more trustworthy than one implying completeness, and it stops the next engineer assuming guarantees that aren't there.
Replacing the engine ran new-alongside-old invisibly, then A/A validation to confirm the false-positive rate matched nominal, then a cutover that froze methodology for in-flight tests. Changing how significance is computed mid-experiment invalidates the experiment.
Experiment definitions live in the docs tool analysts and PMs already use, replicated by an off-the-shelf connector. Zero admin UI to build, zero bespoke sync code, and PMs can review config without warehouse access.