Create a set of benchmark dataset

[dreamquark-ai]

Create a set of benchmark dataset

[athewsey]

@Optimox those figures were I believe all using develop code as of my recent PR #164. Taking a random 80/10/10 Training/Validation/Test split of Forest Cover Type, and just trying the different PyTorch framework versions via AWS' provided deep learning container images on SageMaker - so all with Python 3.6, Ubuntu 16.04, and (if I interpret the container versioning correctly) CUDA 10.1... But there's a chance there are some small, relevant library differences between them. All the training was run on an ml.p3.2xlarge instance, so backed by 1xV100 GPU.

Appreciate the library versions aren't as controlled as they could be between tests, and will try to re-run on a fully controlled/local env with only PyTorch different if possible - but it's tricky as my current workflow is mostly set up for using those pre-built images. Just thought it was worth mentioning for consideration in this ticket's priority, and as I hadn't seen discussion about cross-version benchmarking/accuracy checks elsewhere on the project.

[ddofer]

I can help with this. It'd be best to run this with a testing framework, to allow for the tests or CI to check if changes to the models/code (e.g. defaults or improvements) break/reduce performance

[athewsey]

Anecdotally, I recently noticed a drop in accuracy (or maybe convergence speed) on Forest Cover Type when upgrading version of PyTorch... Would be interested to see whether others experience same and understand whether there's some issue that needs addressing or it's just a statistical variation.

Stopping at 200 epochs, observed test accuracy of:

• 95.584% on PyTorch v1.4.0
• 88.128% on PyTorch v1.5.1
• 92.840% on PyTorch v1.6.0

[Optimox]

@athewsey thanks for reporting that, it seems quite a lot for just changing the torch version. Have you been experimenting this on the latest release just by changing the pytorch version? I understand that random seeds could change from one version to another, but after 200 epochs there should not be such a gap.

@Hartorn @eduardocarvp did you notice such strong changes when monitoring tabnet scores?

[Optimox]

adding some kaggle links:

• https://www.kaggle.com/carlmcbrideellis/tabnet-a-very-simple-regression-example
• https://www.kaggle.com/carlmcbrideellis/tabnet-simple-binary-classification-example