Data leakage
Not met yet
Data leakage is when the training data gives the answer away in a way that won't be there when the model is used for real, so its scores look far better than it will ever do. The commonest leak is a column filled in after the event, like how many tickets were sold, which is only known once the night has turned out busy: ask when each column is known. The other is preprocessing that saw the answers: choosing columns, scaling or filling gaps using every row, test rows included, before cross-validating. Put every step that learns anything inside the pipeline. Cross-validation can't catch either; a score that seems too good is the clue.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a model that scores perfectly with a column filled in after the event will still forecast well before that column is known.
- Thinks cross-validation will notice a column that gives away the answer.
- Thinks choosing the columns that best match the answers, using every row, and then cross-validating gives an honest score.
- Thinks a suspiciously high score is good news rather than a reason to look for a leak.
- Thinks a leaky column would show up as a poor score on the test set, when it is recorded there the same way.