[Reproducing Results] on Alfworld

[ysymyth]

[Reproducing Results] on Alfworld

[ai-nikolai]

@ysymyth - any update on the above? Thank you.

[ysymyth]

Have you tried to run on more than 10 cases? 10 seems too noisy to tell anything.

[ai-nikolai]

@ysymyth Thanks a lot for coming back to me on this one.

Yes, we have run the results several times on the entire test set (i.e. 135 environments). We both used your code and our own implementation.

Did you do modifications beyond the ones present in your code (in this repo?):

• e.g. Correcting common syntax mistakes of the model (e.g. put object x in place y -> put object x in/on place y)?
• Sampling outputs again with increasing temperature?
• Anything else?