API Idea: add a "drop" argument

[GaelVaroquaux]

API Idea: add a "drop" argument

[jeromedockes]

I mean 3 or 4 -- ie having a separate transformer for select. indeed it can do more than just subset columns

[TheooJ]

I agree that it is something that would be useful, and to reply to your points :

I like 2. more than 1. because I think it makes sense to have different dropping strategies for different transformers. That being said, if they share a common, widely used drop case it would make sense to have this argument in TableVectorizer too. This way you would set up the dropping strategy for all of them at once

3. It could possibly account for other strategies than dropna, for instance practitioners often drop 1) very sparse columns, or features that are present only for a small number of ids, 2) correlations (among features, between feature and target), 3) outliers

4. I like the idea of verbs over nouns, but I would choose nouns to avoid a contrast with scikit-learn

wdyt ?

[Vincent-Maladiere]

@TheooJ I think Gaël comment is about removing columns based on user-defined lists. You'd need a transformer to perform feature selection.

Maybe we could combine 1. and 4. : having a drop parameter on the TableVectorizer and allowing renaming or simple column manipulation operations in kwargs. In addition, we could also introduce the ColSelector for usage out of TableVectorizer.

Let's assume a slightly different identity from scikit-learn with verbs rather than nouns ;)

[jeromedockes]

I think I prefer option 3, "add a Drop transformer". The vectorizer has quite a few parameters already and I believe a slightly longer pipeline with simpler steps is easier to understand than a pipeline where some steps do a lot of things. Also, I'm not sure but there could be situations where a user wants control over where the drop happens, eg to drop a lot of columns as soon as possible to save memory, or to use a column for a join and drop it afterwards for prediction