No in-built functionality for tracking of metrics during training

[asappresearch]

No in-built functionality for tracking of metrics during training

[jeremyasapp]

we could also consider just computing the metrics on a per batch level during training and logging that, but then things like dropout will affect the training metrics. That's true in your proposed solutions as well, unless you re thinking of doing this during the eval step?

[cle-ros]

The problem with the per batch level are things like AUC. If we are using the batch as negatives (as is quite common), computing the AUC per batch will be much less accurate than computing it per epoch (and using all samples from an epoch as negatives).

Besides, either approach would allow us to unify this (taken from _eval_step in train.py):

log(f'{tb_prefix}Validation/Loss', val_loss, self._step)
log(f'{tb_prefix}Validation/{self.metric_fn}', val_metric, self._step)
log(f'{tb_prefix}Best/{self.metric_fn}', self._best_metric, self._step)  # type: ignore
for metric_name, metric in self.extra_validation_metrics.items():
    log(f'{tb_prefix}Validation/{metric_name}',
        metric(preds, targets).item(), self._step)  # type: ignore

With either

for metric in self.metrics:
    metric.finalize()
    metric.log(log_func) . # log_func could be any log function, defaulting to the one above

Or

for metric in self.metrics:
    metric.finalize(metrics_state)
    metric.log(log_func, metrics_state)

That has the additional advantage that we would support logging of metrics that are more complex natively. Imagine, e.g., a combined recall-precision-fscore metric, that could jointly log all three. Or one that computes a conditional metric if, say, you have different types of samples. Than it could log things like "accuracy for type1: ..." and "accuracy for type2: ..."

[jeremyasapp]

What do you propose to fo with the training and validationloss over the whole dataset? Since people generally use torch loss objects which won't have the "incremental" logic?