CosineAnnealingLR

[open-mmlab]

CosineAnnealingLR

[HeCheng0625]

Hi, we updated a PR to fix the problem. You can check it! (we use: from diffusers.optimization import get_cosine_schedule_with_warmup)

[lixuyuan102]

Hi, we updated a PR to fix the problem. You can check it! (we use: from diffusers.optimization import get_cosine_schedule_with_warmup)

Thanks for the reply. I'm using the NoamScheduler with a base_lr of 0.05 and 2000 warmup steps to train Valle. Have you tested the differences between NoamScheduler and "cosine_schedule_with_warmup"? If so, could you share which is better and the best parameters? Thanks again and looking for your reply.

[lmxue]

@HeCheng0625 Please follow up on this issue.

[HeCheng0625]

Hi, we haven't test NoamScheduler, I think using AdamW with lr between 5e-5 to 1e-4 and cosine schedule with warmup steps between 5K to 1W steps will give a more stable training process.

[lixuyuan102]

Hi, we haven't test NoamScheduler, I think using AdamW with lr between 5e-5 to 1e-4 and cosine schedule with warmup steps between 5K to 1W steps will give a more stable training process.

Thanks for sharing.