Repeatable runs with Hammurabi simulator

[imagine-consortium]

Repeatable runs with Hammurabi simulator

[shutsch]

The code seems to be parallelized on some level according to my htop, presumably within Hammurabi?
I know numpy also multithreads some calculations, but I do not know if any of those are called within IMAGINE.

The legacy random number generator is indeed known to be not thread-safe, see e.g. this issue here:
https://stackoverflow.com/questions/58875076/numpy-random-seed-and-multithreading-causes-differing-results

But I am no expert on this, this might of course be an issue solely related to Hammurabi.

[shutsch]

We had a similar issue in Nifty, which I think is related to numpy multithreading.

If I explicitly force OMP_NUM_TREADS=1 before executing your code I get:

`------------------------------
Deterministic case

Run 1
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 2115.1283515418586 2115.1283515418586

Run 2
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 2115.1283515418586 2115.1283515418586

Run 3
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 2115.1283515418586 2115.1283515418586

Run 4
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 2115.1283515418586 2115.1283515418586

---

Stochastic case

Run 1
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 5322.912641998637 5423.724789142201

Run 2
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 5322.912641998637 5423.724789142201

Run 3
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 5322.912641998637 5423.724789142201

Run 4
ensemble seeds: 1608637542 1273642419
sync_I_data (sum): 5322.912641998637 5423.724789142201
`

, which is the same between runs.
One might have to switch to the new numpy random generators and not use the legacy np.random.seed() function to avoid this.

[luizfelippesr]

Thanks for spotting this!

The problem, however, is not related to the numpy legacy random number generator, as one can see that the ensemble seeds are the same on every run (note that there is actually nothing happening in parallel in IMAGINE in the above example). These seed values are then written in the parameter file that hammurabiX uses, and hammurabiX itself runs (externally, through a subprocess pipe) in parallel using OpenMP.

Initially, I thought that the problem was caused by IMAGINE writing the seed in the wrong section of the XML parameters file. After your post, I starting to believe that this could be a problem with HammurabiX itself.

@gioacchinowang, do you have any thoughts?

[luizfelippesr]

Fixed by commit hammurabi-dev/hammurabiX@8eef522