Comments (8)
We technically don't thoroughly check that the data doesn't overlap right now. We set a given seed and generate missions, set a different seed and generate a different set of missions in sequence. Would it be acceptable to just say, training data is seeds in the range 1 to 1 million, testing is after 1 million?
Just to be rigorous, you could generate a set of demos that way and I could write a piece of code that checks if, for each demo, the same mission string + initial env config is present in the training set. Probably better than just assuming we're ok.
from babyai.
from babyai.
Let's use this issue to protocol how we generate missions everywhere in the code
- currently, in demonstration generation,
--seed=0
is reserved for the validation set. Other seeds can be used for generating training data, as many seeds as many jobs there is. - in
scripts/train_rl.py
seeds for training environments are selected as follows:
- in
babyai/evaluate.py
thebatch_evaluate
function uses a bunch of seeds starting from10^9
:
- in the same file
evaluate
function just a given environment. Inscripts/evaluate.py
it is given an environment that is seeded with 0. - in
scripts/train_intelligent_expert.py
we reuse the same pool of seeds for evaluating and training. Namely, we takenum_new_demos
of new seeds, evaluate the agent on them, and then reuse them for training.
I guess we could simplify things a bit if we agree to use only seeds >= 10^9 for validation, with the exception of scripts/train_intelligent_expert.py
, in the case of which we can stick to the current approach.
from babyai.
On a related note, I am very curious why we do this in babyai/arguments.py
:
from babyai.
Hmmm, randomizing the seed. I assume I copied that from some other script.
from babyai.
There is a call to evaluate()
in train_intelligent_expert.py
: https://github.com/mila-udem/babyai/blob/master/scripts/train_intelligent_expert.py#L328
As long as this method automatically starts from seeds seeds >= 10^9 we should be ok.
from babyai.
from babyai.
from babyai.
Related Issues (20)
- You cannot pick up or put down objects in manual-mode in enjoy.py HOT 6
- Know full vocabulary apriori. HOT 2
- Impossible missions are sampled? HOT 2
- Question about performance HOT 2
- PutNextLocal HOT 8
- Bot bugs (attribute error: prev_carrying) HOT 2
- Unable to run PPO agent with arch=bow_endpool_res_pixels HOT 2
- What's the package version of gym-minigrid and gym used for the baseline results? HOT 2
- When I run 'BabyAI-PutNext-v0', it did not work after 24 hours' training... HOT 1
- Sampling unsolvable demonstrations
- Mission not accomplished in the last observation
- Error during install: "pip install --editable ." in babyai folder HOT 2
- What does the agent see, in 7x7? -- possible bug in the task? HOT 1
- Get demonstration for 'MiniGrid-MultiRoom-N4-S5-v0' HOT 1
- Headless rendering issue for environments with doors HOT 5
- Missing dependency for scikit-build HOT 4
- AttGRU model is broken + my quick Fix HOT 5
- Visualizing stored "images" HOT 2
- Publish pip package HOT 1
- enjoy.py script seems to expect old PyQT version of gym-minigrid HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from babyai.