Comments (3)
we could also consider just computing the metrics on a per batch level during training and logging that, but then things like dropout will affect the training metrics. That's true in your proposed solutions as well, unless you re thinking of doing this during the eval step?
from flambe.
The problem with the per batch level are things like AUC. If we are using the batch as negatives (as is quite common), computing the AUC per batch will be much less accurate than computing it per epoch (and using all samples from an epoch as negatives).
Besides, either approach would allow us to unify this (taken from _eval_step
in train.py
):
log(f'{tb_prefix}Validation/Loss', val_loss, self._step)
log(f'{tb_prefix}Validation/{self.metric_fn}', val_metric, self._step)
log(f'{tb_prefix}Best/{self.metric_fn}', self._best_metric, self._step) # type: ignore
for metric_name, metric in self.extra_validation_metrics.items():
log(f'{tb_prefix}Validation/{metric_name}',
metric(preds, targets).item(), self._step) # type: ignore
With either
for metric in self.metrics:
metric.finalize()
metric.log(log_func) . # log_func could be any log function, defaulting to the one above
Or
for metric in self.metrics:
metric.finalize(metrics_state)
metric.log(log_func, metrics_state)
That has the additional advantage that we would support logging of metrics that are more complex natively. Imagine, e.g., a combined recall-precision-fscore metric, that could jointly log all three. Or one that computes a conditional metric if, say, you have different types of samples. Than it could log things like "accuracy for type1: ..." and "accuracy for type2: ..."
from flambe.
What do you propose to fo with the training and validationloss over the whole dataset? Since people generally use torch loss objects which won't have the "incremental" logic?
from flambe.
Related Issues (20)
- `Script` should support positional arguments HOT 2
- Cannot add custom attributes to a flambe module HOT 1
- Relative paths no longer work for local resources
- Reporting server not logging requests when tensorboard is installed.
- More useful stage dependency graph
- Check cluster compatibility of the yaml config before starting the cluster
- AvgPooling fails even when input dimensions match expected ones. HOT 1
- Richer debug mode HOT 1
- Separate embedder and encoder in flambe.nn.embedder HOT 3
- Warning when copying a tensor in the base sampler
- Reporting script can die on the orchestrator HOT 1
- Out-of-the-box experiments with standard datasets HOT 1
- Make `extra_validation_metrics` a key-value mapping rather than a list HOT 1
- RNN (sru) is incompatible with torch 1.3
- Make all Flambe base-models torchscript compatible HOT 1
- Allow pickle for saving checkpoints in Experiment HOT 1
- TaggedScalar instead of Schema when using make_component HOT 1
- Early checks for constructor argument incompatibilities HOT 2
- Better data handling for datasets HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flambe.