Paul's Python CoursePPC

A/B tests and what they cost

Not met yet

An A/B test hands each version of something to a share of the people who see it, and compares how often each one works. A plain test splits them evenly and waits until the end, so it keeps handing out the losers right up to the last day. What that costs is the regret: for each version, the people handed it times the gap between its rate and the best rate, added up. Only a simulation knows the true rates, so only a simulation can count regret exactly. A bandit test shifts people towards the likely winner as it learns, which costs far less, but it ends knowing the losers' rates only roughly.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks an even-split A/B test costs nothing because it is fair, when every visitor handed a weaker version is a conversion the test expects to lose.
  • Works out regret as every visitor times the gap between the best and the worst versions, rather than each version's visitors times its own gap to the best.
  • Thinks a bandit test ends knowing every version's rate as precisely as an even split, when it hands the losing versions far fewer visitors.

Where it comes up