Paul's Python CoursePPC

Exploring and exploiting

Not met yet

Exploiting means taking the arm with the best estimate, to do well now. Exploring means trying something else, to learn. Every bandit strategy trades one against the other. Epsilon-greedy explores a fixed fraction epsilon of the time, at random, for ever. UCB explores where it is least sure: it picks the biggest estimate + c * sqrt(ln(total) / count), so an arm tried less gets a bigger bonus. Thompson sampling draws one plausible rate for each arm from Beta(wins + 1, losses + 1) and takes the biggest, so an arm is tried about as often as it might really be the best.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks UCB picks the arm with the highest average reward, ignoring the bonus for arms tried less.
  • Thinks UCB explores by chance, like epsilon-greedy, when its choice is fixed by the estimates and counts.
  • Thinks rng.integers(0, n) can return n, as random.randint would, so writes len(arms) - 1 and never explores the last arm.
  • Thinks epsilon-greedy stops exploring once it has found the best arm, when it goes on exploring a fraction epsilon of the time for ever.
  • Thinks every exploring run is wasted, so the best strategy never explores.
  • Thinks an exploring run never lands on the current best arm, when exploring picks among all the arms.

Where it comes up