macOS conda installation has missing random generators

[nicolossus]

macOS conda installation has missing random generators

[steffengraber]

Hi @nicolossus Thank you for your notification. On which system are you working - Mac or Linux? Which version of python are you using?

[nicolossus]

@steffengraber I am using a conda environment with Python 3.11 on macOS. A minimal reproducer:

conda create --name nest_conda -c conda-forge python=3.11 nest-simulator=3.6

My guess would be that the conda build is compiled without the Random123 generators for some reason.

[steffengraber]

Ok, I tested this under ubuntu 23.04. With this I get

>>> nest.rng_types
('Philox_32', 'Philox_64', 'Threefry_32', 'Threefry_64', 'mt19937', 'mt19937_64')

It seems that there is only a problem in the MacOS packages.

[steffengraber]

The test for nrg.types is now integrated in the CI. These tests run for linux_64 and osx_64, but not for osx-arm64. This allowed me to narrow down the problem to macOS with M1 chip. I do not have the possibilities to test this. Could you possibly verify this? @nicolossus

[nicolossus]

@steffengraber I should probably have mentioned that I encountered the conda installation problem on osx-arm64 (M2 chip), so I can confirm.

However, if I build and install NEST from source on the same osx-arm64 machine, all the RNG types are included. It therefore doesn't seem like there should be a problem with Apple silicon and the Random123 generators library (which contains the Philox and Threefry generators).

[steffengraber]

if I build and install NEST from source on the same osx-arm64 machine, all the RNG types are included

Could you send me a build log of it? This could help to recognize the differences in the build process.

[nicolossus]

@steffengraber Of course, I have sent you the build log directly.

[steffengraber]

The problem might have something to do with cross-compiling the osx_arm64 version.
The report here on random123 :

...
-- Info: Check if Random123 generators work. 
-- Info: Bad CPU type in executable 
-- Info: Check if Random123 generators work. OFF
...

[steffengraber]

@nicolossus As a test, I have now deactivated the random123 check and used this to build the package for macOS ARM.

You can download one here:
https://artprodeus21.artifacts.visualstudio.com/A910fa339-c7c2-46e8-a579-7ea247548706/84710dde-1620-425b-80d0-4cf5baca359d/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbmRhLWZvcmdlL3Byb2plY3RJZC84NDcxMGRkZS0xNjIwLTQyNWItODBkMC00Y2Y1YmFjYTM1OWQvYnVpbGRJZC84MzE4MjEvYXJ0aWZhY3ROYW1lL2NvbmRhX2FydGlmYWN0c18yMDIzMTEyOC4xLjFfb3N4X2FybTY0X251bXB5MS4yM3B5dGhvbjMuMTEuX19fX2NweXRob241/content?format=zip

Can you test this? Unfortunately, I have no possibility to do so and the automatic tests are not executable due to cross compiling.

To test, unzip the package and change into the directory. Then:

conda create -n my-test-env -c "file://${PWD}" -c conda-forge 
conda activate my-test-env
conda install -c "file://${PWD}" nest-simulator

Instead of conda you can also use mamba, which offers some advantages when creating environments.

[nicolossus]

@steffengraber Installing the osx-arm64 package from the build works, and I was able to successfully run the serial pytests from the test suite.

As I understand it, the problem is that osx-arm64 packages are currently built on conda-forge's macOS x86_64 infrastructure. There might not be a solution for the cross-compilation issue we have seen here in the near future... Should we keep this issue open as a reminder or consider the issue resolved?

[steffengraber]

Just added a note to the documentation. With this, we can close the issue. Additionally, I have also informed conda-forge support about the issue.

[heplesser]

@steffengraber Building with the Random123 on osx-arm64 is an interesting idea for experimentation, but for production, your current solution of just shipping the conda package without the Random123 generators on osx-arm64 and documenting that properly is the best solution. I hope the conda-forge folks will provide native osx-arm64 builds at some point. I would not trust random numbers for them Random123 generators unless their cmake-tests have run successfully.

@nicolossus I'd suggest to close this issue now.