Cannot Reproduce the results in the paper.

[yfzhang114]

Cannot Reproduce the results in the paper.

[JeanKaddour]

Thank you very much for your interest in our work, again!

• For the TCGA dataset, we use a bias strength of 0.1 for the results in Section 6.1, and when we tested the robustness to different bias strengths in Section 6.3, we went up to 0.6. I remember our propensity policy being very sensitive to the bias strength for the TCGA covariates, so bias=1.0 might produce something degenerate (e.g., always assigning the same treatment for all covariates). Although, as of now, it is not clear to me why this should hurt SIN's performance, particularly in comparison to the baselines. Would you please try again with lower bias strengths?
• The provided default hyper-parameters should produce sensible results. However, we tuned them over 10 runs with random search and values sampled from the ranges provided in Table 4 in the Appendix to ensure a fair comparison across methods. Also, please note that we averaged the errors in the paper across 10 random seeds.
• The above SIN results look strange; generally speaking, for all methods, the errors should increase as more treatments are added to the test set (i.e., as K increases). Specifically for SIN, the learned propensity features are less informative for test treatments that are very unlikely to be selected for certain covariates.

[yfzhang114]

Thanks for your kind response! I will have a try.

[JeanKaddour]

I just ran the TCGA experiment with all default values, as specified in the argument parser (the default bias = 0.3).
SIN:

Graphite:

[yfzhang114]

For bias=0.3, I got almost the same result as you, however, when setting bias=0.1, the performance is just as I mentioned. 😂

[JeanKaddour]

Thank you for sharing that insight!

• I set up a new machine, installed all libraries from scratch and now, for bias=0.1, I'm actually running into errors. I suspect this to be caused by some numerical issues due to changes in the used libraries or the QM9 dataset. When I ran the experiments more than half a year ago on a different machine, I didn't experience them. I will try to debug this soon, my apologies for that issue.
• For bias=0.2, I get these results with the default hyper-parameters:
  SIN:
  GraphITE:
  These results still deviate from the results reported in the paper; but there we chose the best runs from our hyper-parameter search based on the validation loss. I hope this helps.

[yfzhang114]

Get it! Thanks!

[JeanKaddour]

I found and fixed the runtime error I experienced for TCGA, bias=0.1. The cause was that very few QM9 graphs have a number of edge types <= 1, which is different from the majority with number 3. For some reason, this broke the minibatch graph collation implemented in PyG 2, printing a non-informative exception. I suspect that I didn't experience this back when I ran the experiments almost one year ago because of the switch from PyG1 to PyG2.

Simply running

python generate_data.py
python run_model_training.py

yields the below results.

In-Sample

Out-Sample