TODO: also configure logging for sub-processes(not master)

[DelinQu]

TODO: also configure logging for sub-processes(not master)

[SunQpark]

Hi @DelinQu, thank you for raising this issue.

It seems that get_logger is problematic as you pointed out and I'm currently working on this. I'll let you know if there is any progress.

Also, if you are interested in using hydra_DDP branch of this repo, I would recommend using my clone version of that. I made a few commits which solves serious bug(It was not using DDP at all) a while ago on that branch, but forgot to make a PR to apply that fix to this repo. I will make PR soon, but it could take some time.

[DelinQu]

Hi @DelinQu, thank you for raising this issue.

It seems that get_logger is problematic as you pointed out and I'm currently working on this. I'll let you know if there is any progress.

Also, if you are interested in using hydra_DDP branch of this repo, I would recommend using my clone version of that. I made a few commits which solves serious bug(It was not using DDP at all) a while ago on that branch, but forgot to make a PR to apply that fix to this repo. I will make PR soon, but it could take some time.

Thanks for your replying SunQpark, I will follow your DDP! 😃

[SunQpark]

Oh thanks, but I'll let this issue open yet!

[DelinQu]

Oh thanks, but I'll let this issue open yet!

Hi SunQpark,
your repo has really helped me tremendously, but I got another problem when the training process is early stopped:

Although it doesn't affect my models much, the error persists. My configuration file is as follows:

n_cpu: 8
n_gpu: 8
batch_size: 4
learning_rate: 0.0001
weight_decay: 0
scheduler_step_size: 50
scheduler_gamma: 0.1
status: train
trainer:
  epochs: 500
  logging_step: 100
  monitor: min loss/valid
  save_topk: 5
  early_stop: 10
  tensorboard: true