Comments (9)
There is not enough information to consider you are in the over fitting realm : what is your train vs valid score ? the fact that features change from month to month simply shows that you have data shifts over time, not that it's not reasonable to rely on different features.
You can also set lambda_sparse to 0 to limit sparsity.
You can also limit number of epochs to avoid overfitting.
from tabnet.
thanks, I will investigate further and let you know!
from tabnet.
Fair enough, after plotting loss curves doesn’t look like over-fitting per se.
See below for n_d=n_a=8, n_steps=2, lambda_sparse=e-6
Model is retrained monthly, but while there may be data shifts over time it shouldn’t be too dramatic as there’s only 2% of the time-period changing from one month to the next.
There could be variations from the random customer base used for the validation set, but again this is only 10% of the data. Not clear why this would be so much more erratic for TabNet than XGBoost.
Boxplot below shows feature importance variation on 12 consecutive models
Looking at a few of the most unstable features, feature importance changes look a bit erratic.
Compare this to the same box-plot for XGBoost, features are picked up much more consistently.
Any thoughts on what could be causing these variations.
Many thanks!
from tabnet.
Are you computing feature importance on the training set or the monthly validation set?
Do you have better predictive scores with XGBoost or TabNet ?
Are you sure that all your tabnet models converge correctly before the end of training?
from tabnet.
Hi again @Optimox,
Feature importances computed on training set as per here.
I have slightly better performance with XGBoost on most months.
I had max_epochs=100, patience=60.
Now I've increased it to max_epochs=200 I have a mix of runs ending between 120 and 200 epochs, a few getting all the way to 200, although loss trajectory looks very flat.
Many thanks again!
from tabnet.
Are you using a learning rate scheduler ?
IMO, the best way to train neural networks is to tweak the learning rate and number of epochs so that you don't need to use early stopping anymore. With a good decay and number of epochs your model should reach its best validation score at the last epoch (or very close to best score). I do not know the size of your dataset here, but early stopping could be one explanation for the large differences you see on a monthly basis.
from tabnet.
from tabnet.
Hi again, so, also getting this behaviour using a StepLR scheduler, training to 200 epochs, not using early stopping
from tabnet.
Then I don't know, are you sure you are always feeding your features in the same order and correctly attributing the features and the importance ?
from tabnet.
Related Issues (20)
- TabNetRegressor vs other networks HOT 1
- spike in memory when training ends HOT 8
- Severe overfitting HOT 18
- OOM problem when I search hyperparameters with Tabnet HOT 3
- Support for complex-valued datasets HOT 4
- Different classification variables in the test set and train set HOT 1
- Struggling to get model to fit - Help Wanted HOT 7
- Optimizing TabNet for Disease Classification with Continuous Audio Features HOT 1
- Interpreting Sparsity on Global Importance HOT 5
- ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() HOT 1
- Validation loss HOT 1
- Lightweight Fine-tunning or few-shot learning for limited labeled data HOT 1
- Maybe `drop_last` should be set as False in default? HOT 1
- Incompatiblity of current round() method with pytorch tensors when performing early stopping HOT 1
- Retraining a saved model on different dataset HOT 3
- change device seems not work HOT 8
- when I try to train with pretrained model loaded, kernel dies HOT 2
- Feature
- Multi-label classification - how to start HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tabnet.