Comments (6)
Hi, just to make sure, did you use the models in the docker image with the code that is also in the docker image? We've made changes to the master branch since, so if you're using the models from the docker image with code from the current github repo, it's to be expected that this won't work.
If that's not the problem, can you tell us what shell commands you used to run evaluate.py?
from babyai.
Hi Maxime,
Thanks for the quick reply.
So for the results I have posted I used the models from the docker image with the code from the github repo. But after you pointed it out I used the code provided in the docker image, and the results did not change much. Now I am getting 12% success rate for the GoToSeq-v0 environment, and 69% success rate for the PickupLocal-v0. I have used the same models as mentioned in the earlier post.
I used the following shell commands:
python3 evaluate.py --env BabyAI-GoToSeq-v0 --model GoToSeq-1
python3 evaluate.py --env BabyAI-PickupLoc-v0 --model PickupLoc-4_best
from babyai.
The commands that you provided don't work as-is, you'd have to use:
python3 -m scripts.evaluate --env BabyAI-GoToSeq-v0 --model il_baselines/GoToSeq-1
python3 -m scripts.evaluate --env BabyAI-PickupLoc-v0 --model rl_baselines/PickupLoc-4_best
They seem to be working on my end. Evaluating with just 50 episodes because this laptop has a slow GPU:
root@2f97e72b8915:/babyai# python3 -m scripts.evaluate --env BabyAI-PickupLoc-v0 --model rl_baselines/PickupLoc-4_best --episodes 50
F 362 | FPS 136 | D 0:00:02 | R:xsmM 0.90 0.07 0.70 0.97 | S 1.00 | F:xsmM 7.2 4.9 2 21
10 worst episodes:
- episode 29: R=0.7046874999999999, F=21
- episode 45: R=0.7046874999999999, F=21
- episode 25: R=0.7328125, F=19
- episode 2: R=0.775, F=16
- episode 46: R=0.775, F=16
- episode 20: R=0.803125, F=14
- episode 15: R=0.8171875, F=13
- episode 10: R=0.83125, F=12
- episode 17: R=0.83125, F=12
- episode 40: R=0.83125, F=12
root@2f97e72b8915:/babyai# python3 -m scripts.evaluate --env BabyAI-GoToSeq-v0 --model il_baselines/GoToSeq-1 --episodes 50
F 7813 | FPS 240 | D 0:00:32 | R:xsmM 0.86 0.16 0.00 1.00 | S 0.98 | F:xsmM 156.3 193.7 2 1152
10 worst episodes:
- episode 28: R=0.0, F=1152
- episode 22: R=0.5640624999999999, F=279
- episode 38: R=0.5757812499999999, F=543
- episode 2: R=0.675, F=416
- episode 36: R=0.7015625, F=191
- episode 46: R=0.7234375, F=177
- episode 19: R=0.7375, F=168
- episode 5: R=0.746875, F=162
- episode 15: R=0.759375, F=154
- episode 32: R=0.759765625, F=615
Do you have nvidia-docker
installed, or are you using plain docker?
from babyai.
The reason I had used the commands like that was because I was running them from the script directory, and had kind of remapped the model directory such that it worked like that.
But anyways I tried to do it the way you suggested. So if I use the shell commands while in the docker image they work fine.
But earlier I had just copied all the stuff out of the docker container into my local machine, and was running it like that. So even now when I tried to run the shell commands you have mentioned on my local copy, it still gives those not so good results I have mentioned earlier.
I am not sure why is it happening. Is it something to do with running everything inside the docker container? Sorry I have never used docker before so not sure about it.
from babyai.
The docker images also contains an installation of torch, python, numpy, etc. It maintains everything in a "frozen" configuration so the results stay replicable, eg: you're running exactly the setup we has when we trained those models. It's hard to say exactly what's different with your configuration outside of the docker container.
from babyai.
I did not know that. Seems like that might have been the issue then.
Thanks for the help!
from babyai.
Related Issues (20)
- You cannot pick up or put down objects in manual-mode in enjoy.py HOT 6
- Know full vocabulary apriori. HOT 2
- Impossible missions are sampled? HOT 2
- Question about performance HOT 2
- PutNextLocal HOT 8
- Bot bugs (attribute error: prev_carrying) HOT 2
- Unable to run PPO agent with arch=bow_endpool_res_pixels HOT 2
- What's the package version of gym-minigrid and gym used for the baseline results? HOT 2
- When I run 'BabyAI-PutNext-v0', it did not work after 24 hours' training... HOT 1
- Sampling unsolvable demonstrations
- Mission not accomplished in the last observation
- Error during install: "pip install --editable ." in babyai folder HOT 2
- What does the agent see, in 7x7? -- possible bug in the task? HOT 1
- Get demonstration for 'MiniGrid-MultiRoom-N4-S5-v0' HOT 1
- Headless rendering issue for environments with doors HOT 5
- Missing dependency for scikit-build HOT 4
- AttGRU model is broken + my quick Fix HOT 5
- Visualizing stored "images" HOT 2
- Publish pip package HOT 1
- enjoy.py script seems to expect old PyQT version of gym-minigrid HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from babyai.