Comments (19)
@Catypad, compatibility with sklearn is not established at the level of using sklearn pipelines as in GridSearchCV
out of the box. You may try ParameterGrid
as in the pseudocode below:
from sklearn.model_selection import ParameterGrid
# Function that instantiates a tabnet model.
def create_tabnet(n_d=32, n_steps=5, lr=0.02, gamma=1.5,
n_independent=2, n_shared=2, lambda_sparse=1e-4,
momentum=0.3, clip_value=2.):
model = TabNetClassifier(
n_d=n_d, n_a=n_a, n_steps=n_steps,
lr=lr,
gamma=gamma, n_independent=n_independent, n_shared=n_shared,
cat_idxs=cat_idxs,
cat_dims=cat_dims,
cat_emb_dim=cat_emb_dim,
lambda_sparse=lambda_sparse, momentum=momentum, clip_value=clip_value,
optimizer_fn=torch.optim.Adam,
scheduler_params = {"gamma": 0.95,
"step_size": 20},
scheduler_fn=torch.optim.lr_scheduler.StepLR, epsilon=1e-15, verbose = 0
)
return model
# Generate the parameter grid.
param_grid = dict(n_d = [24, 32],
n_a = [24],
n_steps = [3, 4, 5],
lr = [0.01, 0.02],
gamma = [1, 1.5, 2],
lambda_sparse = [1e-2, 1e-3, 1e-4],
momentum = [0.3, 0.4, 0.5],
n_shared = [2],
n_independent = [2],
clip_value = [2.],
)
grid = ParameterGrid(param_grid)
search_results = pd.DataFrame()
for params in grid:
params['n_a'] = params['n_d'] # n_a=n_d always per the paper
tabnet = create_tabnet()
tabnet.set_params(**params)
tabnet.fit(X_train=X_train, y_train=y_train,
X_valid=X_valid, y_valid=y_valid,
max_epochs=num_epochs, patience=patience,
batch_size=batch_size, virtual_batch_size=virtual_batch_size,
)
y_prob = tabnet.predict_proba(X_test)
score = accuracy_score(y_test, y_prob)
results = pd.DataFrame([params])
results['score'] = np.round(score, 3)
search_results = search_results.append(results)
Hope it helps!
from tabnet.
We might be missing a self.params
, it would not cost much to add it!
from tabnet.
I hope my feedback has been received as constructive, as it's meant to be.
That said, I'll do my best to add it, cheers!
from tabnet.
this shoud be fixed by #84
from tabnet.
Hey just saw this comment. I implemented and tested inheriting get_params
and set_params
from sklearn.base.BaseEstimator
. It appears to pass an ok repr, not sure if you want to use the one already implemented in the classifier.
I appreciate this is now anyway fixed in a different commit but should pushing my fork be helpful let me know. Many thanks
from tabnet.
The model seems to break now further down when calling .fit()
AttributeError: 'NoneType' object has no attribute 'shape'
(full error below). A similar error is thrown by sklnear's fit if a X_valid
is passed.
Possibly due to incompatible args in the .fit()
of sklearn estimators (e.g. GridSearchCV) and the .fit()
in tab_model.py
.
I may be able to look into this next week.
.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
AttributeError: 'NoneType' object has no attribute 'shape'
FitFailedWarning)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-6-d0a805e488c2> in <module>()
----> 1 grid_search = grid_search.fit(X_train, y_train)
.local/lib/python3.6/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
737 refit_start_time = time.time()
738 if y is not None:
--> 739 self.best_estimator_.fit(X, y, **fit_params)
740 else:
741 self.best_estimator_.fit(X, **fit_params)
.local/lib/python3.6/site-packages/pytorch_tabnet/tab_model.py in fit(self, X_train, y_train, X_valid, y_valid, loss_fn, weights, max_epochs, patience, batch_size, virtual_batch_size)
106
107 self.update_fit_params(X_train, y_train, X_valid, y_valid, loss_fn,
--> 108 weights, max_epochs, patience, batch_size, virtual_batch_size)
109
110 train_dataloader, valid_dataloader = self.construct_loaders(X_train,
.local/lib/python3.6/site-packages/pytorch_tabnet/tab_model.py in update_fit_params(self, X_train, y_train, X_valid, y_valid, loss_fn, weights, max_epochs, patience, batch_size, virtual_batch_size)
421 else:
422 self.loss_fn = loss_fn
--> 423 assert X_train.shape[1] == X_valid.shape[1], "Dimension mismatch X_train X_valid"
424 self.input_dim = X_train.shape[1]
425
AttributeError: 'NoneType' object has no attribute 'shape'
from tabnet.
Hello @meechos
Not sure what you are trying to do, if you could share a snippet of code.
In case you are trying to use sklearn Grid search, it will not work with early stopping up front
You can do it in several ways to make it work :
- use ParameterSampler instead, and keep best params and model after each iteration.
- build a simple wrapper around the classifier and give it to the grid search
Here is an example for LGBM I used in some notebook, you can adapt it.
The important is that in the fit, you do the split and give X_valid and Y_valid
Tell me if you need more help on this
from lightgbm import LGBMClassifier
class MyLGBMClassifier(LGBMClassifier):
def predict(self, *args, **kwargs):
return self.predict_proba(*args, **kwargs)
def fit(self, X, Y):
X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.1, random_state=42, shuffle=True, stratify=Y)
return super().fit(X_train, Y_train, eval_metric='logloss', verbose=100, early_stopping_rounds=5, eval_set=[(X_valid, Y_valid)])
from tabnet.
Hi @Hartorn , @meechos ! I'm trying to use Grid Search and I'm having the same problem and just with the information above I can't resolve it. Can you help me, please? I don't have a lot of experience!
from tabnet.
@meechos @Catypad Indeed, such as with XGB or other, you need to use ParameterGrid, or ParameterSampler, or if you want to use GridSearch you can make a wrapper, as described up.
from tabnet.
@Hartorn just gave your solution a try and getting the same issues reported earlier above. I am aware it works generally, but not sure if you have tried the suggested wrapper method with TabNetClassifier
.
EDIT: copied correct error log
--------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-4e204e88670c> in <module>()
----> 1 grid.fit(X_train, y_train)
/home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
737 refit_start_time = time.time()
738 if y is not None:
--> 739 self.best_estimator_.fit(X, y, **fit_params)
740 else:
741 self.best_estimator_.fit(X, **fit_params)
<ipython-input-1-b919ab2c7351> in fit(self, X, Y)
50 def fit(self, X, Y):
51 X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
---> 52 return super().fit(X_train, Y_train, eval_metric='average_precision', verbose=100, early_stopping_rounds=30, eval_set=[(X_valid, Y_valid)])
53
54
TypeError: fit() got an unexpected keyword argument 'eval_metric'
from tabnet.
Can you share a notebook so I can see?
from tabnet.
Sure thanks for your reply. See here a modification on the current forest example notebook.
EDIT: notebook bugfix
from tabnet.
@meechos your TabnetTuner should be
class TabNetTuner(TabNetClassifier):
def predict(self, *args, **kwargs):
return self.predict_proba(*args, **kwargs)
def fit(self, X, Y):
X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
return super().fit(X_train, Y_train, patience=30, X_valid=X_valid, y_valid=Y_valid)
You used some arguments that the classifier was not accepting : eval_set, verbose, early_stopping_rounds are boosting model propoerties, here you needed to use X_valid, y_valid, and patience
from tabnet.
Thank you for your help!
from tabnet.
@Hartorn thanks, of course. I tried that and still breaks.
Device used : cuda
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-17-4d8fc662f890> in <module>()
46
47 grid = GridSearchCV(clf2, param_grid=param_grid, scoring='f1')
---> 48 grid.fit(X_train, y_train, X_valid, y_valid)
TypeError: fit() takes from 2 to 4 positional arguments but 5 were given
And/or
FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
NameError: name 'patience' is not defined
FitFailedWarning)
Device used : cuda
Device used : cuda
Device used : cuda
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-20-39987737f9df> in <module>()
46
47 grid = GridSearchCV(clf2, param_grid=param_grid, scoring='f1')
---> 48 grid.fit(X_train, y_train)
1 frames
<ipython-input-20-39987737f9df> in fit(self, X, Y)
15 def fit(self, X, Y):
16 X_train, X_valid, Y_train, Y_valid = train_test_split(X, Y, test_size=0.15, random_state=42, shuffle=True, stratify=Y)
---> 17 return super().fit(X_train, Y_train, X_valid, y_valid, patience)
18
19 clf2 = TabNetTuner(
NameError: name 'patience' is not defined
from tabnet.
@meechos Second one is the way to do it.
I guess you did not define a patience
variable
Plus you need to have a look at what the classifier is accepting as params
https://github.com/dreamquark-ai/tabnet/blob/develop/pytorch_tabnet/tab_model.py#L76
As I showed on the the code extract in my previous comment, you need to name the arguments in fit patience=30, X_valid=X_valid, y_valid=Y_valid
fit is taking more arguments than this, so if you are using them as positional one, you need to give everyone of them, or use named one as in my example.
from tabnet.
@Catypad No worries, are you managing it ?
from tabnet.
@Catypad @meechos Here is a example of a notebook doing a grid search, hope this help :)
Cheers
from tabnet.
Would be very helpful to see it implemented thank you @Hartorn.
Mum said never download from strangers :) Why not share it in the examples, surely many people would be interested in parameter optimisation.
from tabnet.
Related Issues (20)
- TabNet overfits (help wanted, not a bug) HOT 9
- TabNetRegressor vs other networks HOT 1
- spike in memory when training ends HOT 8
- Severe overfitting HOT 18
- OOM problem when I search hyperparameters with Tabnet HOT 3
- Support for complex-valued datasets HOT 4
- Different classification variables in the test set and train set HOT 1
- Struggling to get model to fit - Help Wanted HOT 7
- Optimizing TabNet for Disease Classification with Continuous Audio Features HOT 1
- Interpreting Sparsity on Global Importance HOT 5
- ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() HOT 1
- Validation loss HOT 1
- Lightweight Fine-tunning or few-shot learning for limited labeled data HOT 1
- Maybe `drop_last` should be set as False in default? HOT 1
- Incompatiblity of current round() method with pytorch tensors when performing early stopping HOT 1
- Retraining a saved model on different dataset HOT 3
- change device seems not work HOT 8
- when I try to train with pretrained model loaded, kernel dies HOT 2
- Feature
- Multi-label classification - how to start HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tabnet.