reseed the rng for every mission when generating demos

[mila-iqia]

reseed the rng for every mission when generating demos

[maximecb]

We technically don't thoroughly check that the data doesn't overlap right now. We set a given seed and generate missions, set a different seed and generate a different set of missions in sequence. Would it be acceptable to just say, training data is seeds in the range 1 to 1 million, testing is after 1 million?

Just to be rigorous, you could generate a set of demos that way and I could write a piece of code that checks if, for each demo, the same mission string + initial env config is present in the training set. Probably better than just assuming we're ok.

[rizar]

Given how complex the levels are, I believe it would be sufficient to always use different seeds for training and testing. You could verify this with a script if you want. I am going through the code right now and checking our usage of random seeds. Most likely, your suggestion is OK, although I would go for a bit more future-proof pool of training missions, reserving seeds from 1 to 10^9 for training.

…

On Tue, 22 Jan 2019 at 10:10 Maxime Chevalier-Boisvert < ***@***.***> wrote: We technically don't thoroughly check that the data doesn't overlap right now. We set a given seed and generate missions, set a different seed and generate a different set of missions in sequence. Would it be acceptable to just say, training data is seeds in the range 1 to 1 million, testing is after 1 million? Just to be rigorous, you could generate a set of demos that way and I could write a piece of code that checks if, for each demo, the same mission string + initial env config is present in the training set. Probably better than just assuming we're ok. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8Ym5hsY2u0IrUGcDoVKCRmnUcHh_Bks5vFynxgaJpZM4aM0-X> .

[rizar]

Let's use this issue to protocol how we generate missions everywhere in the code

• currently, in demonstration generation, --seed=0 is reserved for the validation set. Other seeds can be used for generating training data, as many seeds as many jobs there is.
• in scripts/train_rl.py seeds for training environments are selected as follows:
  
• in babyai/evaluate.py the batch_evaluate function uses a bunch of seeds starting from 10^9:
  
• in the same file evaluate function just a given environment. In scripts/evaluate.py it is given an environment that is seeded with 0.
• in scripts/train_intelligent_expert.py we reuse the same pool of seeds for evaluating and training. Namely, we take num_new_demos of new seeds, evaluate the agent on them, and then reuse them for training.

I guess we could simplify things a bit if we agree to use only seeds >= 10^9 for validation, with the exception of scripts/train_intelligent_expert.py, in the case of which we can stick to the current approach.

[rizar]

On a related note, I am very curious why we do this in babyai/arguments.py:

[maximecb]

Hmmm, randomizing the seed. I assume I copied that from some other script.

[maximecb]

There is a call to evaluate() in train_intelligent_expert.py: https://github.com/mila-udem/babyai/blob/master/scripts/train_intelligent_expert.py#L328

As long as this method automatically starts from seeds seeds >= 10^9 we should be ok.

[rizar]

The line you are referring to calls `babyai.imitation.ImitationLearning.validate()`, which in turns calls `evaluate()` with `args.val_seed`, which should be equal to `int(1e9)`, cause this is its default value.

…

On Tue, 22 Jan 2019 at 15:00 Maxime Chevalier-Boisvert < ***@***.***> wrote: There is a call to evaluate() in train_intelligent_expert.py: https://github.com/mila-udem/babyai/blob/master/scripts/train_intelligent_expert.py#L328 As long as this method automatically starts from seeds seeds >= 10^9 we should be ok. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YlcD6TAJl_5qNV4VJuMH2v1qrY-fks5vF23qgaJpZM4aM0-X> .

[rizar]

Fixed by #25 and #24