Comments (5)
@Optimox those figures were I believe all using develop code as of my recent PR #164. Taking a random 80/10/10 Training/Validation/Test split of Forest Cover Type, and just trying the different PyTorch framework versions via AWS' provided deep learning container images on SageMaker - so all with Python 3.6, Ubuntu 16.04, and (if I interpret the container versioning correctly) CUDA 10.1... But there's a chance there are some small, relevant library differences between them. All the training was run on an ml.p3.2xlarge
instance, so backed by 1xV100 GPU.
Appreciate the library versions aren't as controlled as they could be between tests, and will try to re-run on a fully controlled/local env with only PyTorch different if possible - but it's tricky as my current workflow is mostly set up for using those pre-built images. Just thought it was worth mentioning for consideration in this ticket's priority, and as I hadn't seen discussion about cross-version benchmarking/accuracy checks elsewhere on the project.
from tabnet.
I can help with this. It'd be best to run this with a testing framework, to allow for the tests or CI to check if changes to the models/code (e.g. defaults or improvements) break/reduce performance
from tabnet.
Anecdotally, I recently noticed a drop in accuracy (or maybe convergence speed) on Forest Cover Type when upgrading version of PyTorch... Would be interested to see whether others experience same and understand whether there's some issue that needs addressing or it's just a statistical variation.
Stopping at 200 epochs, observed test accuracy of:
- 95.584% on PyTorch v1.4.0
- 88.128% on PyTorch v1.5.1
- 92.840% on PyTorch v1.6.0
from tabnet.
@athewsey thanks for reporting that, it seems quite a lot for just changing the torch version. Have you been experimenting this on the latest release just by changing the pytorch version? I understand that random seeds could change from one version to another, but after 200 epochs there should not be such a gap.
@Hartorn @eduardocarvp did you notice such strong changes when monitoring tabnet scores?
from tabnet.
adding some kaggle links:
- https://www.kaggle.com/carlmcbrideellis/tabnet-a-very-simple-regression-example
- https://www.kaggle.com/carlmcbrideellis/tabnet-simple-binary-classification-example
from tabnet.
Related Issues (20)
- The mask tensor M in script tab_network.py needs to be transformed to realize the objective stated in the paper: "γ is a relaxation parameter – when γ = 1, a feature is enforced to be used only at one decision step".
- Current version on conda-forge is 4.0 while 4.1 is already released HOT 8
- Minimal working example for TabNetRegressor/Classifier HOT 4
- Transfer learning, capability to change structure of model HOT 1
- Generate Embeddings for Tabular Data HOT 1
- TabNet overfits (help wanted, not a bug) HOT 9
- TabNetRegressor vs other networks HOT 1
- spike in memory when training ends HOT 8
- Severe overfitting HOT 18
- OOM problem when I search hyperparameters with Tabnet HOT 3
- Support for complex-valued datasets HOT 4
- Different classification variables in the test set and train set HOT 1
- Struggling to get model to fit - Help Wanted HOT 7
- Optimizing TabNet for Disease Classification with Continuous Audio Features HOT 1
- Interpreting Sparsity on Global Importance HOT 5
- ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() HOT 1
- Validation loss HOT 1
- Lightweight Fine-tunning or few-shot learning for limited labeled data HOT 1
- Maybe `drop_last` should be set as False in default? HOT 1
- Incompatiblity of current round() method with pytorch tensors when performing early stopping HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tabnet.