Permutation tests and p-values
Not met yet
A permutation test asks whether a difference between two groups could be luck. Measure the real difference; then suppose the labels didn't matter, pool the readings, shuffle them with a seeded rng.permutation, deal them back into groups of the same sizes, and measure the difference again, thousands of times. The p-value is the share of shuffles that do at least as well as the real one. A small p-value says luck rarely makes a gap that big; it doesn't say how big the effect is, or why. A large one says luck could easily have done it, so the data can't tell, which is not the same as proving there's no difference.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Counts only the shuffles whose difference is strictly bigger than the real one. The real arrangement is one of the possible shuffles, so a p-value counts those at least as big.
- Thinks the p-value is the chance that the claim is true, or that the difference is real, rather than how often luck alone would make a difference at least as big.
- Thinks a large p-value proves there is no difference. It only means luck could easily have made the difference seen, so the data can't tell.
- Thinks a tiny p-value means a large difference. It means luck rarely matches the difference; with enough readings, even a small difference gets a tiny p-value.