Everybody quotes 95% and almost nobody states what it is 95% about. It is not the chance your variant is better. It is a statement about how often this procedure would mislead you if there were no difference at all.
Every term used above is defined in our experimentation glossary, and three experiments taken apart step by step sit in the case studies. Longer pieces with the workings attached are in the field notes.
It says: if the two versions were identical, a result this extreme would appear by chance about 5% of the time. It says nothing on its own about how likely your variant is to be better, because that also depends on how often your ideas are right in the first place.
Do this: Stop reading significance as «95% chance the variant wins». Read it as «this would happen by luck one time in twenty».
The maths is built on a sample size chosen before launch and a single decision at the end. Checking daily and stopping when it crosses gives the effect many chances to appear by luck.
Do this: Fix your sample size before launch and look once, or use a sequential method designed for repeated looks.
Send traffic and baseline rate. We reply with the smallest effect your test can actually detect. One email, no call.
With enough traffic a 0.1% difference becomes significant. Significance tells you an effect probably exists. It does not tell you the effect is worth the engineering time.
Do this: Decide the smallest lift worth shipping before the test. Compare the result against that, not only against the p-value.
An interval running from minus 2% to plus 9% is honest about what you learned, which is «probably not harmful, possibly useful». The single word «significant» hides all of that.
Do this: Report the interval alongside the verdict. It changes what people do with the result.
This is teaching material and our own reading of standard practice, not advice for your specific site. Check anything important with your own specialist before you act on it.
Traffic, baseline rate and what you changed. We reply with the minimum detectable effect and whether the result meant anything. No cost, no call.