sotelo/scribe

Out of pencils? No problem! I've got you covered.

scribe's Issues

Decomposing Adasecant

Hey @caglar!

Maybe we can coordinate here to make sure that the changes I did were the right ones. I ran an experiment with these changes, but if there's something wrong (probably there is) I can kill the experiments and re run.

Variance Reduction:

• The code already has an option: use_corrected_grad that allows to disable this component.

Adagrad:

• The code already has an option: use_adagrad that allows to disable this component.

Normalization (Block-wise normalization of the gradients):

• I added the option use_block_normalization to control this. I only conditioned the line 315. A lot of quantities are dependent on this. So I am not sure if changing more things is needed.

Extra: Outlier Gradient detection
Do you think we should also compare with and without the outlier detection? I added an option used in line 461.

Am I missing something? In our talk a couple of weeks ago, you told me to deactivate Adasecant, not Adagrad. After seeing the code, I thought you were referring to Adagrad, not Adasecant. Please let me know if I am mistaken, to fix the problem.

Cheers!
Jose