We write these for ourselves first. Every number has a source, every method has the workings attached, and when we got something wrong we say so in the note rather than quietly editing it.
The measurement took twenty minutes. Checking the measurement took forty, and it moved the headline number from 46% to 25%. I am writing up the mistake rather than the finding, because the mistake is the more useful of the two.
eBay ran the experiment, published it, and the gap between the dashboard and the truth was about forty times. Then I sized a holdout for a normal advertiser and found most of them cannot run one.
A 5% lift on checkout completion needs about 15,000 sessions per variant. At a thousand checkout starts a month that is two and a half years, which is longer than the checkout will exist.
The false positive rate is supposed to sit at 5%. I measured what happens to it when somebody opens the dashboard every morning, and the honest answer is 27.44%. Then I found out that a bigger site does not help.