drkwint/pyroclast

TensorFlow-based library for PhD research

pyroclast's Issues

Make hyperparameters explicit in a defaults file

Put hyperparameters in their own defaults file so that they are clearly visible and each experiment can refer easily to the defaults when noting what the experimental variables are.

Implement Huang et al.'s BoostResNet

Add Interpretable mixin

The interpretable mixin should provide basic functionality for retrieving data used for basic visualizations. This may be a bad name for the mixin and better names may be discussed here.

Further, the interpretable mixin needs to be tested to ensure we are able to retrieve accurate and usable data.

Implement TrainingStrategy

Implement a system for training an ensemble, mostly for boosting, with different strategies (e.g., layer wise training, e2ebp, or chain training)

CpVAE isn't generating anything that looks like data

Training with default parameters and the MNIST dataset produces generated data that looks randomish. Distortion loss is decreasing reasonably, but images are still bad.

Create a dataset which preprocesses images with a conv stack

Is your feature request related to a problem? Please describe.
Want to ensure that large conv stacks don't need to be run repeatedly during training when they're not being trained.

Implement basic prototype visualization in ProtoPNet

Add total accuracy calculations to train and test for ProtoPNet

Is your feature request related to a problem? Please describe.
Want to have access to a decisive number which measures accuracy over the entire train/test set rather than just TB (which is visual, but imprecise).

Describe the solution you'd like
I think it could just be written to command line during the training process.

Ensure that the equint/cpvae branch is passing CI tests

Write BoostResNet Multi-class loss fn

See Appendix F of Huang et al.'s paper for details

alpha_0 should not be trainable

Lines 30 and 31 in build_graph

Implement FeatureClassifier for Prototype

Is your feature request related to a problem? Please describe.
Implement the common.FeatureClassifier mixin for the Prototype class

Describe the solution you'd like
Update the CpVAE class to inherit the mixin class

Add phase 2 and phase 3 training to prototype

Error in running boost_resnet on Cifar 10

Errors during running: tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[31] = [31, 3] does not index into param shape [1,10]

Ran with: python -m pyroclast.eager_run --alg boost_resnet --dataset cifar10
commit 3a071a2

The network appears to be functioning correctly, and outputs were as expected.

Implement FeatureClassifier interface

Implement the FeatureClassifier interface / mixin. A model extending this base class should break up the model flow into:

1. Retrieving features. Example: A Conv stack evaluated to find features before passing through dense layer(s).
2. Classification using features: Once we have the features, actually performing classification.

By splitting up a model into these two sections, we can perform an analysis of the features that make up classification, including usefulness and robustness.

Write a test for CpVAE training

Write a test that ensures that CpVAE will run without error for 3 forward/backward passes on MNIST to always ensure that changes don't break the basic operation of the model.

Correlation not implemented correctly in FeatureClassifier

Describe the bug
We made the same assumptions as in the paper and that doesn't hold empirically.

Implement SmoothGrad

SmoothGrad is a very simple, but effective technique for computing saliency maps. Given an input x, it generates samples around x by adding Gaussian noise and then computing the saliency map on the sample. Then, it averages the saliency maps together.

Implement Spatial Broadcast Decoder for VAEs

From the paper Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs out of DeepMind, it should just improve VAE models orthogonal to just about any other change.

Update feature_classifier.usefulness and robustness to work on iterators rather than datasets

Is your feature request related to a problem? Please describe.
Makes it a pain in the ass to work in graph mode in a .map

Describe the solution you'd like
See title

register() for custom dataset

As we discussed, it would be great to have a module that can register one's custom dataset into the network.

Create a fresh version of requirements.txt

Wonky Self-Boosting Behavior

COMMIT: 590e7b8

CMD LINE: python -m pyroclast.run

ISSUE:
It looks like something about how the accuracy is calculated is busted. It looks like it should be accurate for the first module, but as soon as the second starts training, it goes to zero
trial_stdout.txt

Add a Continuous Bernoulli output distribution to the C+VAE for MNIST

Paper: https://arxiv.org/abs/1907.06845
Reddit discussion: https://www.reddit.com/r/MachineLearning/comments/ce9xce/r_the_continuous_bernoulli_fixing_a_pervasive/

Update repo README with venv and tensorboard

Describe the bug
README is missing any info on venv or how to visualize in tensorboard. Also, there are some small correctness issues that should be updated.

Make M2's $p(y)$ distribution more general

In particular, we're talking about wanting to use a product of Bernoulli for p(y) rather than a discrete distribution.

To do:

1. Characterize requirements of a distribution (e.g., reparameterized? etc.)
2. Add documentation on requirements
3. Change code to take an input distribution
4. Provide an example with product Bernoulli

Upgrade the calculation of robustness to use an iterative attack

Currently FeatureClassifier.robustness uses the fast_gradient_method to calculate an adversarial perturbation. This means that the attack sometimes fails to actually work (i.e., decrease the model output it intends to decrease). We should upgrade to an iterative attack that is more reliable.

Phase 2 of training ProtoPNet reduces classification accuracy to chance

And phase 3 training doesn't raise the classification rate back to where it was before.

My hypothesis is that there is some problem with how I implemented the prototype push in phase 2.

Screencaps coming eventually for visual evidence of this issue (and maybe a tensorboard log file)

Use Python Loggers

Problem
We currently use print statements everywhere to show what is happening during training. Print go directly to stdout and can be clobbered in multiprocessing settings and have numerous other issues. Further, there is no way to disable logs in production modes or only show logs that describe errors. There is no granularity in the amount of information presented to you.

Solution
Use Python Logging. Python logging is thread safe and allows for variable information control. Further, I believe that we can output the log levels to different files (not sure on this one).

Implement FeatureClassifier for CpVAE

Is your feature request related to a problem? Please describe.
Implement the common.FeatureClassifier mixin for the CpVAE class

Describe the solution you'd like
Update the CpVAE class to inherit the mixin class

Implement BoostDistributionStrategy

Implement a modular system for plugging in algorithms for calculating a boosting distribution over multiple rounds of boosting.