vacancy/pdsketch-alpha-release

pdsketch-alpha-release's Issues

Details of the baselines implemented in the original paper.

Hello! Thank you for providing this valuable code repository.
Currently, I am trying to reproduce the baseline experiment described in the PDSketch paper [https://arxiv.org/abs/2303.05501]. I have followed the instructions in the paper and implemented the training and testing process.
For the BC baseline, I initially used a three-layer Fully Connected Neural Network to encode the world states and a single linear layer to predict the action. However, the generalization performance was quite poor (in-distribution success rate: 1.0; object-generalization success rate: <0.2) and significantly underperformed the GNN-based results in the paper(in-distribution success rate: 0.9; object-generalization success rate: >0.7). I noticed that your implementation used a two-layer graph neural network and mentioned permutation invariance in the paper. Unfortunately, the paper does not provide specific implementation details.
Could you please provide some details about the GNN used in the baseline, such as the construction of the graph input or any relevant references used for its implementation?
I would greatly appreciate any guidance or assistance you can provide to help me overcome this issue. Thank you for your time, and I look forward to your reply!

About reproducing the results from the original paper.

Thanks for such an interesting work.

We are trying to reproduce the results of the original paper based on the provided code, such as Table 1 (BabyAI results) in the main paper [https://arxiv.org/abs/2303.05501].

However, we find that the current training and validation scripts seem to be implemented for the "go to an X" task, which is different from the three tasks mentioned in the original paper: "go to an X", "pick up an X", and "open an X". So, how should I reproduce the results in Table 1 of the paper? Should I train and evaluate the three tasks separately, or is it necessary to use curriculum learning, starting from simpler tasks and gradually progressing to harder tasks, and then evaluating on the three tasks? Would you mind providing the script to reproduce Table 1? It may help us better understand the training approach of PDSketch. If I missed any key details from the original paper, please let me know. Thank you very much, and I look forward to your response.