Conflict between bf16-mixed Precision Setting and MegatronHalfPrecisionPlugin in MegatronGPT Training

[moutasemalakkad]

Conflict between bf16-mixed Precision Setting and MegatronHalfPrecisionPlugin in MegatronGPT Training

[analogtechnica]

@moutasemalakkad

Hi, I've just faced same issue.

• work on Docker image ; nvcr.io/nvidia/nemo:24.03.framework

Here is my solution : insert cfg.trainer.precision = None above this line.

NeMo/examples/nlp/language_modeling/megatron_gpt_continue_training.py

Line 167 in ebba8b1

trainer = Trainer(plugins=plugins, strategy=strategy, **cfg.trainer, callbacks=callbacks)

This solution is inspired from this PR #8908

It should solve the conflict.

[moutasemalakkad]

Thanks! That also did not work, the work around was to set the plugins to an empty list

trainer = Trainer(plugins=[], strategy=strategy, **cfg.trainer, callbacks=callbacks)