RandomizedSearchCV with pytorch-tabnet

[dreamquark-ai]

RandomizedSearchCV with pytorch-tabnet

[meechos]

@Catypad, compatibility with sklearn is not established at the level of using sklearn pipelines as in GridSearchCV out of the box. You may try ParameterGrid as in the pseudocode below:

from sklearn.model_selection import ParameterGrid

# Function that instantiates a tabnet model.
def create_tabnet(n_d=32, n_steps=5, lr=0.02, gamma=1.5, 
                  n_independent=2, n_shared=2, lambda_sparse=1e-4, 
                  momentum=0.3, clip_value=2.):
    
    model = TabNetClassifier(
        n_d=n_d, n_a=n_a, n_steps=n_steps,
        lr=lr,
        gamma=gamma, n_independent=n_independent, n_shared=n_shared,
        cat_idxs=cat_idxs,
        cat_dims=cat_dims,
        cat_emb_dim=cat_emb_dim,
        lambda_sparse=lambda_sparse, momentum=momentum, clip_value=clip_value,
        optimizer_fn=torch.optim.Adam,
        scheduler_params = {"gamma": 0.95,
                         "step_size": 20},
        scheduler_fn=torch.optim.lr_scheduler.StepLR, epsilon=1e-15, verbose = 0
    )
    return model
                  
# Generate the parameter grid.
param_grid = dict(n_d = [24, 32],
                  n_a = [24],
                  n_steps = [3, 4, 5],
                  lr = [0.01, 0.02],
                  gamma = [1, 1.5, 2],
                  lambda_sparse = [1e-2, 1e-3, 1e-4],
                  momentum = [0.3, 0.4, 0.5],
                  n_shared = [2],
                  n_independent = [2],
                  clip_value = [2.],     
)

grid = ParameterGrid(param_grid)

search_results = pd.DataFrame() 
for params in grid:
    params['n_a'] = params['n_d'] # n_a=n_d always per the paper
    tabnet = create_tabnet()
    tabnet.set_params(**params)
    tabnet.fit(X_train=X_train, y_train=y_train,
               X_valid=X_valid, y_valid=y_valid,
               max_epochs=num_epochs, patience=patience,
               batch_size=batch_size, virtual_batch_size=virtual_batch_size,
              )

    y_prob = tabnet.predict_proba(X_test)
    score = accuracy_score(y_test, y_prob)
    
    results = pd.DataFrame([params])
    results['score'] = np.round(score, 3)
    search_results = search_results.append(results)

Hope it helps!

[Optimox]

We might be missing a self.params, it would not cost much to add it!

[meechos]

I hope my feedback has been received as constructive, as it's meant to be.

That said, I'll do my best to add it, cheers!

[Optimox]

this shoud be fixed by #84

[meechos]

Hey just saw this comment. I implemented and tested inheriting get_params and set_params from sklearn.base.BaseEstimator. It appears to pass an ok repr, not sure if you want to use the one already implemented in the classifier.

I appreciate this is now anyway fixed in a different commit but should pushing my fork be helpful let me know. Many thanks

[meechos]

The model seems to break now further down when calling .fit() AttributeError: 'NoneType' object has no attribute 'shape' (full error below). A similar error is thrown by sklnear's fit if a X_valid is passed.

Possibly due to incompatible args in the .fit() of sklearn estimators (e.g. GridSearchCV) and the .fit() in tab_model.py.

I may be able to look into this next week.

---

.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: 
AttributeError: 'NoneType' object has no attribute 'shape'
FitFailedWarning)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-d0a805e488c2> in <module>()
----> 1 grid_search = grid_search.fit(X_train, y_train)

.local/lib/python3.6/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    737             refit_start_time = time.time()
    738             if y is not None:
--> 739                 self.best_estimator_.fit(X, y, **fit_params)
    740             else:
    741                 self.best_estimator_.fit(X, **fit_params)

.local/lib/python3.6/site-packages/pytorch_tabnet/tab_model.py in fit(self, X_train, y_train, X_valid, y_valid, loss_fn, weights, max_epochs, patience, batch_size, virtual_batch_size)
    106 
    107         self.update_fit_params(X_train, y_train, X_valid, y_valid, loss_fn,
--> 108                                weights, max_epochs, patience, batch_size, virtual_batch_size)
    109 
    110         train_dataloader, valid_dataloader = self.construct_loaders(X_train,

.local/lib/python3.6/site-packages/pytorch_tabnet/tab_model.py in update_fit_params(self, X_train, y_train, X_valid, y_valid, loss_fn, weights, max_epochs, patience, batch_size, virtual_batch_size)
    421         else:
    422             self.loss_fn = loss_fn
--> 423         assert X_train.shape[1] == X_valid.shape[1], "Dimension mismatch X_train X_valid"
    424         self.input_dim = X_train.shape[1]
    425 

AttributeError: 'NoneType' object has no attribute 'shape'

[Hartorn]

Hello @meechos

Not sure what you are trying to do, if you could share a snippet of code.
In case you are trying to use sklearn Grid search, it will not work with early stopping up front

You can do it in several ways to make it work :

• use ParameterSampler instead, and keep best params and model after each iteration.
• build a simple wrapper around the classifier and give it to the grid search

Here is an example for LGBM I used in some notebook, you can adapt it.
The important is that in the fit, you do the split and give X_valid and Y_valid
Tell me if you need more help on this

from lightgbm import LGBMClassifier
class MyLGBMClassifier(LGBMClassifier):

    def predict(self, *args, **kwargs):
        return self.predict_proba(*args, **kwargs)
    
    def fit(self, X, Y):
        X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.1, random_state=42, shuffle=True, stratify=Y)
        return super().fit(X_train, Y_train, eval_metric='logloss', verbose=100, early_stopping_rounds=5, eval_set=[(X_valid, Y_valid)])

[Catypad]

Hi @Hartorn , @meechos ! I'm trying to use Grid Search and I'm having the same problem and just with the information above I can't resolve it. Can you help me, please? I don't have a lot of experience!

[Hartorn]

@meechos @Catypad Indeed, such as with XGB or other, you need to use ParameterGrid, or ParameterSampler, or if you want to use GridSearch you can make a wrapper, as described up.

[meechos]

@Hartorn just gave your solution a try and getting the same issues reported earlier above. I am aware it works generally, but not sure if you have tried the suggested wrapper method with TabNetClassifier.

EDIT: copied correct error log

--------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-4e204e88670c> in <module>()
----> 1 grid.fit(X_train, y_train)

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    737             refit_start_time = time.time()
    738             if y is not None:
--> 739                 self.best_estimator_.fit(X, y, **fit_params)
    740             else:
    741                 self.best_estimator_.fit(X, **fit_params)

<ipython-input-1-b919ab2c7351> in fit(self, X, Y)
     50     def fit(self, X, Y):
     51         X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
---> 52         return super().fit(X_train, Y_train, eval_metric='average_precision', verbose=100, early_stopping_rounds=30, eval_set=[(X_valid, Y_valid)])
     53 
     54 

TypeError: fit() got an unexpected keyword argument 'eval_metric'

[Hartorn]

Can you share a notebook so I can see?

[meechos]

Sure thanks for your reply. See here a modification on the current forest example notebook.

EDIT: notebook bugfix

[Hartorn]

@meechos your TabnetTuner should be

class TabNetTuner(TabNetClassifier):

    def predict(self, *args, **kwargs):
        return self.predict_proba(*args, **kwargs)
    
    def fit(self, X, Y):
        X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
        return super().fit(X_train, Y_train, patience=30, X_valid=X_valid, y_valid=Y_valid)

You used some arguments that the classifier was not accepting : eval_set, verbose, early_stopping_rounds are boosting model propoerties, here you needed to use X_valid, y_valid, and patience

[Catypad]

Thank you for your help!

[meechos]

@Hartorn thanks, of course. I tried that and still breaks.

Device used : cuda
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-4d8fc662f890> in <module>()
     46 
     47 grid = GridSearchCV(clf2, param_grid=param_grid, scoring='f1')
---> 48 grid.fit(X_train, y_train, X_valid, y_valid)

TypeError: fit() takes from 2 to 4 positional arguments but 5 were given

And/or

FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: 
NameError: name 'patience' is not defined

  FitFailedWarning)
Device used : cuda
Device used : cuda
Device used : cuda
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-20-39987737f9df> in <module>()
     46 
     47 grid = GridSearchCV(clf2, param_grid=param_grid, scoring='f1')
---> 48 grid.fit(X_train, y_train)

1 frames
<ipython-input-20-39987737f9df> in fit(self, X, Y)
     15     def fit(self, X, Y):
     16         X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
---> 17         return super().fit(X_train, Y_train, X_valid, y_valid, patience)
     18 
     19 clf2 = TabNetTuner(

NameError: name 'patience' is not defined

[Hartorn]

@meechos Second one is the way to do it.
I guess you did not define a patience variable

Plus you need to have a look at what the classifier is accepting as params
https://github.com/dreamquark-ai/tabnet/blob/develop/pytorch_tabnet/tab_model.py#L76
As I showed on the the code extract in my previous comment, you need to name the arguments in fit patience=30, X_valid=X_valid, y_valid=Y_valid
fit is taking more arguments than this, so if you are using them as positional one, you need to give everyone of them, or use named one as in my example.

[Hartorn]

@Catypad No worries, are you managing it ?

[Hartorn]

@Catypad @meechos Here is a example of a notebook doing a grid search, hope this help :)
Cheers

Téléchargements.zip

[meechos]

Would be very helpful to see it implemented thank you @Hartorn.

Mum said never download from strangers :) Why not share it in the examples, surely many people would be interested in parameter optimisation.