Reproducing separation

[greenelab]

Reproducing separation

[gwaybio]

Hi @zalky

Thanks for your interest in the paper and code! The code to reproduce figure 3B is in scripts/viz/feature_activation_plots.R and is discussed in #65 . Essentially, I believe that the size of the points given above obscures separation.

At a more fundamental level, the sample activation scores (for example, the ones given in encoded_rnaseq_onehidden_warmup_batchnorm.tsv) are inherently unstable. The results depend on random initialization conditions prior to training. We are working on this issue currently. But for now, it would be incorrect to train a different Tybalt model and think that the numerical designation of the latent encodings are consistent.

Thanks!
Greg

[zalky]

Hi Greg, thanks for the response!

I agree that given the non-deterministic training model you would see some variation in results, but I was a bit surprised by the degree of difference. However, I think I have a lead on what is going on:

I was loading the encoded data in encoded_rnaseq_onehidden_warmup_batchnorm.tsv with:

encoded_df = pd.read_table(encoded_file, index_col=0)

As is generally seen throughout the python code. Unfortunately I could not get this to reproduce the figures. However, if you load the encoded data without specifying the index column:

encoded_rnaseq_df = pd.read_table(encoded_file)

Then I can successfully reproduce Fig. 3B exactly (sans clinic colours) for encoding 53 and 66:

But by not specifying index_col=0, this inserts the sample labels as the first encoding. When plotting one encoding against another, this means the encoding numbers will be off by one.

Going back, specifying index_col=0, and re-plotting encoding 52 vs 65 re-produces Fig. 3B. Re-training the model from scratch also produces figures much more in line with the paper, as long as you take into account the off by one encoding numbers.

I haven't pursued this issue any further than this, but is it possible that somewhere, maybe in the R code, something may be loading the sample labels in the first encoding column, thereby resulting in the encoding labels being off by one?

[gwaybio]

Glad this was figured out - I agree that this issue is a potential pitfall in analyses (see #86)- I will bump this up in priority. Thanks again!