Handle numerical missing values in TableVectorizer

[tomMoral]

Handle numerical missing values in TableVectorizer

[GaelVaroquaux]

And, besides, it's not very hard to write:

make_pipeline(TableVectorizer(), SimpleImputer(), RandomForestClassifier())

Not much more difficult than:

make_pipeline(TableVectorizer(), RandomForestClassifier())

[jeromedockes]

still the goal of the tablevectorizer is to prepare a table so that the rest of the pipeline will work on it without problems, quite a few estimators lack support for missing values, and missing values are ubiquitous, so it is worth trying to find ways to improve the user experience and I would suggest keeping the issue open for discussion

[GaelVaroquaux]

I disagree with your desire to have an option to do it automatically: there is no good default and it tends to depend a lot on the downstream estimator.

If you really want good behavior by default, you should really use HistGradientBoosting, which is very robust to many thing.

[GaelVaroquaux]

This is not a bug in TableVectorizer: it's down to the learner to handle missing values (because the strategy to handle missing values must differ depending on the learner).

If the learning does not handle missing values, you should add an imputer (as you did)

In addition, RandomForests handle missing values in the upcoming release of scikit-learn: scikit-learn/scikit-learn#5870
So your specific problem will disappear real soon.

However, we recommend using HistGradientBoosting avec RandomForest it often works better.

[jeromedockes]

But I agree that at least the default should probably be to output nans where there are missing values as is currently the case

[tomMoral]

I agree that this depends on the downstream classifier but I think having an option to "fill missing value" would be a nice feature as the goal of TableVectorizer is to take a table and "vectorize" it. (that is why this is a feature request and not a bug ;) )

[tomMoral]

Yes it is easy to fix (I used numerical_transformer=SimpleImputer) but I found this behavior unexpected as I thought (without reading the doc) that I would get a vector out of this Transformer.
I find the name confusing as this does not vectorize the table (to me, a vector should have a consistent type for all its entries).
This class only acts on the categories and not the numerical values so maybe it would be better to call it CategoryVectorizer, to make it clear it does not touch the numerics.

my 2cts :)

[GaelVaroquaux]

This class only acts on the categories and not the numerical values so maybe it would be better to call it CategoryVectorizer, to make it clear it does not touch the numerics.

It also deals with the dates