Paul's Python CoursePPC

Cleaning a table

Not met yet

Clean in a fixed order, on a copy: tidy the text with .str.strip() and .str.title(), turn text into numbers with pd.to_numeric, read dates with pd.to_datetime(column, format=..., errors="coerce") one format at a time, fix impossible values with .loc[condition, column] = value, and drop duplicates last, once the copies look alike. Each step hands back something new, so assign it back. And df[rows]["col"] = value changes a copy, never df.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks df[mask]["col"] = value changes the table. The first [] makes a new table, and the change lands on that.
  • Thinks drop_duplicates() catches rows that differ only by a space, a capital letter or a date format.
  • Thinks df["col"].str.strip() changes the column, without assigning the result back.
  • Thinks pd.to_datetime() reads a column of mixed date formats correctly on its own.

Where it comes up