How to impute using simple imputer (custom function) — Tooling guide

mike_plumbs

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and replace with mode replace +infs with max and -infs with min use standard scaler. then for using xgboost i would like to: simply replace -infs/+infs with very large or -ve large numbers. i have been playing with sklearn pipeline and i would like to know...

Originally asked by Maths12 on Stack Exchange (datascience) (CC BY-SA 4.0).

Variant 1071.

uphilltech

Great question — answering as the UphillTech team:

Since you want to save the training min/max and use those to replace inf's in the test set, you need a custom transformer . To build a robust transformer, you should use some of sklearn's validation functions. And it's best to work in numpy, since as you point out an earlier transformer in a pipeline will have already converted an input dataframe to an nparray. (You could stick with dataframes, and either convert in your transformer (losing some efficiency), or make sure your transformer always comes first in a pipeline (alright if you're the only one using your code).) Then, here's a simple version. I'm opting not to have the numeric_features as input; you're probably better off using a...

Full answer by Ben Reiniger: https://datascience.stackexchange.com/a/75064 (CC BY-SA 4.0).

Want it for your business? UphillTech.