A question on beta hyperparameter

[sea-snell]

A question on beta hyperparameter

[Sea-Snell]

Hi, Thanks for the question! Yes, in those configs it is just set incorrectly. Usually I set it to some value in the range 1-16; you can try different values by increasing powers of 2. Also note that beta is an inference time parameter, so you can try different values without having to re-train. Let me know if this clears things up for you, or if you have other questions.

…

On Jul 7, 2023, at 7:44 AM, Denis Tarasov ***@***.***> wrote: Hello, thanks for your work! I was going through the code and we have this line for advantage: w_values = torch.exp(self.beta * (qs - vs)) (https://github.com/Sea-Snell/Implicit-Language-Q-Learning/blob/4af8c5c12baf69c743dc9753e377a427109b7e93/src/models/iql_model.py#L329C15-L329C15) And the beta parameter is supposed to control the impact of reweighing. But in all configs beta is set to 0 https://github.com/Sea-Snell/Implicit-Language-Q-Learning/blob/4af8c5c12baf69c743dc9753e377a427109b7e93/config/wordle/train_iql.yaml#L23 https://github.com/Sea-Snell/Implicit-Language-Q-Learning/blob/4af8c5c12baf69c743dc9753e377a427109b7e93/config/toxicity/train_iql.yaml#L43 https://github.com/Sea-Snell/Implicit-Language-Q-Learning/blob/4af8c5c12baf69c743dc9753e377a427109b7e93/config/vis_dial/train_iql.yaml#L35 Which means that reweighting does not affect output anyhow which is not what we want I think. Am I missing something or is it just incorrect values in configs? Default value of beta is 1 for the model and in paper you report using other values for that paramerter, while configs set it to 0. — Reply to this email directly, view it on GitHub <#5>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSY2WIHX2E5VCLX3FDHUFDXPAODFANCNFSM6AAAAAA2B5F3I4>. You are receiving this because you are subscribed to this thread.

[DT6A]

Thank you very much for your answer! And is there any other location where correct parameters are written down? E.g. in those files I mention gamma seems to be set improperly too (it's value 0.99, not 1.0). Or should I just rely on parameters reported in the paper?

[Sea-Snell]

Go with the values in the paper

[DT6A]

Ok, thank you once again!