Rectified Adam

[amaiya]

Rectified Adam

[amaiya]

Since ktrain currently allows you to use any "Adam-like" optimizer, this is already supported. Once you load a model (either by building it yourself or loading one of ktrain's pre-canned model), simply call model.compile and supply RectifiedAdam as the optimizer. Since RectifiedAdam uses the same hyperparameters as Adam (beta_1 and beta_2), the 1cycle and triangular learning rate policies with cyclical momentum should still work correctly.

For an example, see this Google Colab notebook that uses ktrain with Recified Adam.

[xaerincl]

Does the model have to be a sequential one like in your example?

Im using a pre trained mobilenetv2 from keras and training it with radam and autofit with no problem but when i do this:

loss, acc = learner.model.evaluate_generator(learner.val_data)
i get this error:
ValueError: steps=None is only valid for a generator based on the keras.utils.Sequence class. Please specify steps or use the keras.utils.Sequence class.

So what im doing is using some values for steps(like 1 or 10) but they get differents results for final loss and final accuracy.

[amaiya]

No, the model does not have to be Sequential. It can be any Keras model.

There's a strange issue with Keras where DirectoryFrameIterator is considered an instance of Sequence in some environments (e.g., Python on Ubuntu 18.04), but not in other environments (conda Python on Google Colab, Kaggle). As a result, the steps argument to evaluate_generator and predict_generator is required on Google Colab and Kaggle, but not in, say, your typical local Ubuntu 18.04 installation.

The following workaround should get learner.model.evaluate_generator to work correctly for you (assuming your validation data is in the form of a DirectoryIterator):

learner.model.evaluate_generator(learner.val_data, steps=len(learner.val_data))

The value len(learner.val_data) should return the number of steps or batches in the validation set. I tested the above on Google Colab and it works.

As an aside, I would also recommend upgrading ktrain to the latest version to ensure latest bug fixes are obtained.

[xaerincl]

Thanks! that works

Last question: Any plan to add discriminative learning rates similar to the slice function of fast ai?

[amaiya]

Great. Adding discriminative learning rates to ktrain is under consideration. But, it's currently at a lower priority compared to other planned features, since the significance of the impact is unclear to me right now. I know that with pretrained BERT models, at least, discriminative learning rates seem to have minimal impact.