stevenyangyj/emma-alfworld

Official code for the paper: Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

emma-alfworld's Issues

什么时候开源dagger-dpo的代码？

你好，我非常关注这篇论文，它是多模态大模型从认知走向决策的一个比较有探索意义的科研项目。我很想亲自实验这个算法，害怕自己实现的dagger-dpo存在问题。请问什么时候能开源dagger-dpo算法呢，如果完整代码的开源有困难的话，我也希望能有机会尽早看到dagger-dpo算法核心代码，非常感谢你的贡献。

Details about VLM baselines

Thanks for sharing your great work!
I have a few questions about your work, especially regarding the baselines.

1. Did you fine-tune the VLMs reported in Table 1? I got confused because Section 3.2 primarily mentioned EMMA for the training details.
2. Could you share the prompt you used? I tried to generate action sequences with instructBLIP, but it did not work well for me.

Also, I am looking forward to your code release!

How did VLM get object id from the pixel observation?

Hello. It's a nice paper and thank you for releasing the model code.

I have a question about your environment setting. From your paper and your sft data, the inputs of the VLM for each step only includes task description and the pixel observation, and the action space for the VLM agent is something like "pick up pan 2". However, I am confused about how can VLM know the object IDs from the pixel observation (e.g. why does it know the pan in the pixel observation is pan 2?) Or does the VLM agent also get observation in the text?

Thank you in advance.

When will the code be open?

When will the code be open sourced?