How to log metrics during training

[awslabs]

How to log metrics during training

[gianlucadetommaso]

The training and validation metrics and loss values are all saved into the status when you do

status = prob_model.train(...)

Perhaps is this what you need? I am not sure if I understand the use case of going inside the trainer yet.

[gianlucadetommaso]

If it is only accuracy, loss, or other metrics that only depend on the predictions and data, you can do this in FitMonitor. Something like this:

from fotuna.prob_model import FitConfig, FitMonitor
from fortuna.metric.classification import accuracy
fit_config = FitConfig(monitor=FitMonitor(metrics=(accuracy,)))

You can also pass arbitrary user-defined metrics, as long as it has the same signature as accuracy. The training loss should be logged automatically. The validation loss will be logged if you pass validation data and set the logger to INFO level.

About the callback, yes, look at FitCallback. This is also part of the FitConfig objects. You can do something like

fit_config = FitConfig(
    monitor=FitMonitor(metrics=(accuracy,)),
    callbacks=[FitCallback(...)]
)

Look here to check how FitCallback works. You can set a custom callback either at train_epoch_end, or train_epoch_start, or train_step_end. Anyway, if all you want is accuracy and loss, you won't need this.

[PaulScemama]

Okay great, thank you! I see that the loss/accuracy are printed during training, but I'm unsure how I can go inside the training loop and, for every epoch, do something like wandb.log({"val_loss": val_loss"}), etc.

Any ideas?

[PaulScemama]

@gianlucadetommaso yes this is good!