Comments (14)
@prigoyal I ended up getting Tensorboard working in Colab with a couple edits:
- Load the notebook extension with
%load_ext tensorboard
- Edit the config in the training command from
config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
toconfig.HOOKS.TENSORBOARD_SETUP.USE_TENSORBOARD=true
from vissl.
Hi @prigoyal I am following the tutorial "Understanding VISSL Training and YAML Config" and I have the exact same issue as @Tylersuard.
I have tried what you suggested but I was not able to solve it. I have not changed any code from the original notebook.
Thank you!
from vissl.
I'm seeing the same issue in two different notebooks - Train SimCLR on 1 gpu.ipynb and Understanding VISSL Training and YAML Config.
It looks like the models aren't being trained. After running python3 run_distributed_engines.py ...
, the tensorboard directory is the last info that's logged. Then the command exits.
If I remove the config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
line from the command, running the command gets a little farther. Then I see the error:
INFO 2021-04-07 03:45:31,394 util.py: 241: Broadcasting checkpoint loaded from
Traceback (most recent call last):
File "run_distributed_engines.py", line 194, in <module>
hydra_main(overrides=overrides)
File "run_distributed_engines.py", line 179, in hydra_main
hook_generator=default_hook_generator,
File "run_distributed_engines.py", line 123, in launch_distributed
hook_generator=hook_generator,
File "run_distributed_engines.py", line 166, in _distributed_worker
process_main(cfg, dist_run_id, local_rank=local_rank, node_id=node_id)
File "run_distributed_engines.py", line 159, in process_main
hook_generator=hook_generator,
File "/usr/local/lib/python3.7/dist-packages/vissl/engines/train.py", line 102, in train_main
trainer.train()
File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/trainer_main.py", line 155, in train
self.task.prepare(pin_memory=self.cfg.DATA.PIN_MEMORY)
File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 634, in prepare
self.base_model = self._build_model()
File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 462, in _build_model
model = self._restore_model_weights(model)
File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 399, in _restore_model_weights
append_prefix=append_prefix,
File "/usr/local/lib/python3.7/dist-packages/vissl/utils/checkpoint.py", line 404, in init_model_from_weights
state_dict_key_name in state_dict.keys()
AttributeError: 'NoneType' object has no attribute 'keys'
from vissl.
Hi @prigoyal I finally was able to run the training successfully. I send you my comments to help you further debug the issue:
- The alternative proposed to update the
fvcore
dependency by runningpip install --progress-bar off --upgrade iopath
did not work for me. - What worked for me is the fix you proposed in #248: changing line https://github.com/facebookresearch/vissl/blob/master/vissl/trainer/train_task.py#L464
- All what I discussed only works iff
config.TENSORBOARD_SETUP.USE_TENSORBOARD
is set tofalse
. Otherwise (even with the fix), the script gets stuck in the point mentioned by @Tylersuard. Therefore, there is definitely an issue there.
Hope it helps! And thank you for your support!
from vissl.
thank you for pointing this out. VISSL has evolved since the v0.1.5 package release. We will update the tutorials to reflect this change. :)
You're welcome! I'm happy I could help :) And thanks for updating the tutorials so quickly
from vissl.
Hi @Tylersuard , thank you for reaching out . This is quite weird and from the logs above, I can't spot anything immediate. One possibility: the colab server probably didn't start or got disconnected. Would it be possible to retry the workflow again and verify ? (maybe try a few different tutorials).
Alternatively, are you able to build VISSL from source and use that: https://github.com/facebookresearch/vissl/blob/stable/INSTALL.md#install-from-source-in-pip-environment If that works on your machine, we can rule out whether its a VISSL issue or simple a colab issue :)
from vissl.
I'm seeing the same issue in two different notebooks - Train SimCLR on 1 gpu.ipynb and Understanding VISSL Training and YAML Config.
It looks like the models aren't being trained. After running
python3 run_distributed_engines.py ...
, the tensorboard directory is the last info that's logged. Then the command exits.If I remove the
config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
line from the command, running the command gets a little farther. Then I see the error:INFO 2021-04-07 03:45:31,394 util.py: 241: Broadcasting checkpoint loaded from Traceback (most recent call last): File "run_distributed_engines.py", line 194, in <module> hydra_main(overrides=overrides) File "run_distributed_engines.py", line 179, in hydra_main hook_generator=default_hook_generator, File "run_distributed_engines.py", line 123, in launch_distributed hook_generator=hook_generator, File "run_distributed_engines.py", line 166, in _distributed_worker process_main(cfg, dist_run_id, local_rank=local_rank, node_id=node_id) File "run_distributed_engines.py", line 159, in process_main hook_generator=hook_generator, File "/usr/local/lib/python3.7/dist-packages/vissl/engines/train.py", line 102, in train_main trainer.train() File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/trainer_main.py", line 155, in train self.task.prepare(pin_memory=self.cfg.DATA.PIN_MEMORY) File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 634, in prepare self.base_model = self._build_model() File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 462, in _build_model model = self._restore_model_weights(model) File "/usr/local/lib/python3.7/dist-packages/vissl/trainer/train_task.py", line 399, in _restore_model_weights append_prefix=append_prefix, File "/usr/local/lib/python3.7/dist-packages/vissl/utils/checkpoint.py", line 404, in init_model_from_weights state_dict_key_name in state_dict.keys() AttributeError: 'NoneType' object has no attribute 'keys'
thank you @mkolodny , the issue of AttributeError: 'NoneType' object has no attribute 'keys' comes from fvcore
dependency. Please look at the #248 (comment) and also alternate is to run pip install --progress-bar off --upgrade iopath
which should update the dependency. If the error doesn't go away, do let me know. We will need to report this to the fvcore team.
from vissl.
Hi @prigoyal I am following the tutorial "Understanding VISSL Training and YAML Config" and I have the exact same issue as @Tylersuard.
I have tried what you suggested but I was not able to solve it. I have not changed any code from the original notebook.
Thank you!
thank you @pcanas and @mkolodny , my guess is that the stall could be coming from the tensorboard logging. For debugging, can you try to set config.TENSORBOARD_SETUP.USE_TENSORBOARD=false
and see if that works? This will allow us to narrow down the issue and look into it further.
from vissl.
thank you @pcanas ,
- we will release new conda/pip packages of vissl in June which should help when using packages of vissl
- great to hear
- I'll investigate the tensorboard issue.
from vissl.
Thanks for offering to investigate the tensorboard issue @prigoyal
In case it's helpful to anyone, here's a version of the "Understanding VISSL Training and YAML Config" colab notebook that I was able to get working without tensorboard:
Understanding VISSL Training and YAML Config
(I also had to install the master branch of ClassyVision - that install is included in the linked colab notebook)
from vissl.
Thanks for offering to investigate the tensorboard issue @prigoyal
In case it's helpful to anyone, here's a version of the "Understanding VISSL Training and YAML Config" colab notebook that I was able to get working without tensorboard:
Understanding VISSL Training and YAML Config
(I also had to install the master branch of ClassyVision - that install is included in the linked colab notebook)
thank you for providing the notebook. It seems like you are training VISSL master in which case we do mandate installing the Classy Vision master https://github.com/facebookresearch/vissl/blob/master/INSTALL.md#step-4-install-vissl
from vissl.
@prigoyal I ended up getting Tensorboard working in Colab with a couple edits:
thank you @mkolodny , that is super amazing to hear.
- Load the notebook extension with
%load_ext tensorboard
I went ahead and updated all the colab notebook tutorials (updated the master tutorials)
- Edit the config in the training command from
config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
toconfig.HOOKS.TENSORBOARD_SETUP.USE_TENSORBOARD=true
thank you for pointing this out. VISSL has evolved since the v0.1.5 package release. We will update the tutorials to reflect this change. :)
from vissl.
closing this task as the actionable items have been completed. Summarizing below:
- Load the notebook extension with
%load_ext tensorboard
-> tutorials updated - Edit the config in the training command from
config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
toconfig.HOOKS.TENSORBOARD_SETUP.USE_TENSORBOARD=true
-> tutorials updated - For NoneType error, see #257 (comment) and this will be included in next VISSL package release.
thank you @mkolodny and everyone. Please feel free to open a new issue or reopen this issue in case of any follow-ups / questions :)
from vissl.
closing this task as the actionable items have been completed. Summarizing below:
- Load the notebook extension with
%load_ext tensorboard
-> tutorials updated- Edit the config in the training command from
config.TENSORBOARD_SETUP.USE_TENSORBOARD=true
toconfig.HOOKS.TENSORBOARD_SETUP.USE_TENSORBOARD=true
-> tutorials updated- For NoneType error, see #257 (comment) and this will be included in next VISSL package release.
thank you @mkolodny and everyone. Please feel free to open a new issue or reopen this issue in case of any follow-ups / questions :)
I'm running the Colab tutorials on master and I'm still having the problems described in this issue. The above comment says the tutorials have been updated, but where have they been updated? On Github, it says the tutorials were last updated 2 months ago.
from vissl.
Related Issues (20)
- Unsupervised pre-training on custom data HOT 2
- Example of using ZeRO and LARC
- Incosistent results when running inference on trained resnet model HOT 1
- Low performance of Supervised training
- MLP_DIM in ViT is not adjustable through the config. HOT 2
- [Feature] Online linear probing
- Linear Image classification weights not loading properly: vissl pretrained models simclr or Dcv2 HOT 3
- "on_forward"-hook operations and back-propagation
- EMA does not work on fp16 and does not copy weights?
- First-class support for timm models
- Can't load pretrained seer/swav checkpoints (RG_Y_128GF) HOT 1
- Is it possible to get full imagenet pretrained weights of MoCo v2 ? HOT 2
- NPID + MoCoV2 weights are the same? HOT 1
- Potential confusion over licensing comment
- Does the MoCo implementation do shuffled Batch Norm when run on a single GPU? HOT 3
- Loading TRAINED models with VISSL HOT 9
- Use pretrain outside of vissl
- can someone share/has anyone shared a checkpoint ( with the class head) of a ssl model on imagenet?
- What config relate to learning rate warm up, weight decay, and momentum in 1 node n GPUs (n > 1 && n < 8) config?
- AttributeError: module 'importlib_resources' has no attribute 'is_resource'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vissl.