Missing data
Not met yet
pandas marks a missing number as NaN and a missing date as NaT, and pd.to_numeric(column, errors="coerce") turns text that isn't a number into NaN. NaN isn't equal to anything, not even itself, so find gaps with isna() and count them with isna().sum(). pandas' mean() and sum() skip missing values; NumPy's don't. dropna() leaves gaps out and fillna(value) fills them in, which changes your answers, so a gap is often more honest than a guess.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks NaN == NaN is True, so == np.nan finds the missing values.
- Thinks comparing a value with NaN raises an error, rather than quietly giving False.
- Thinks pandas' mean() counts missing values, as zero or by giving NaN. It skips them.
- Thinks NumPy's mean() skips NaN the way pandas does. One NaN makes NumPy's answer NaN.
- Thinks pd.to_numeric() quietly skips text it can't read. Without errors="coerce" it stops with a ValueError.