Paul's Python CoursePPC

Missing data

Not met yet

pandas marks a missing number as NaN and a missing date as NaT, and pd.to_numeric(column, errors="coerce") turns text that isn't a number into NaN. NaN isn't equal to anything, not even itself, so find gaps with isna() and count them with isna().sum(). pandas' mean() and sum() skip missing values; NumPy's don't. dropna() leaves gaps out and fillna(value) fills them in, which changes your answers, so a gap is often more honest than a guess.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks NaN == NaN is True, so == np.nan finds the missing values.
  • Thinks comparing a value with NaN raises an error, rather than quietly giving False.
  • Thinks pandas' mean() counts missing values, as zero or by giving NaN. It skips them.
  • Thinks NumPy's mean() skips NaN the way pandas does. One NaN makes NumPy's answer NaN.
  • Thinks pd.to_numeric() quietly skips text it can't read. Without errors="coerce" it stops with a ValueError.

Where it comes up