installing on macos in presence of other pytorch instances

[nilsbecker]

installing on macos in presence of other pytorch instances

[LaurentMazare]

I think it will do the right thing as long as LIBTORCH is properly set, so if you add this to your bashrc/profile, then I would hope for opam install torch to work as well as further compilation of libraries that use this package (though a bit tricky for me to actually test)

[LaurentMazare]

Well that's actually what the discover script tries to do but sadly fails at because of the mixed environments. I'm not sure whether there is an easy way to prevent the C++ compiler from seeing the system wide install here (without preventing it from seeing the files that it actually needs to see), I'll try googling around.

[nilsbecker]

i'm happy to test fixes but unfortunately, am not an expert on how the macos toolchain locates include files etc.

[nilsbecker]

to try to get closer to the issue i am trying to build manually, by git cloning and make all. i get a similar error, but this time actually because of a mixup of two installed pytorches: anaconda and homebrew.

i then try to set LIBTORCH to the homebrew installed pytorch in order to avoid the mixup. so far no luck. here is what i have tried:

# find /usr/local/ -name torch.h
/usr/local//Cellar/pytorch/1.13.1/include/torch/csrc/api/include/torch/torch.h
# LIBTORCH=/usr/local/Cellar/pytorch/1.13.1/include/torch/csrc/api/include make all
File "src/wrapper/dune", line 5, characters 13-22:
5 |   (cxx_names torch_api)
                 ^^^^^^^^^
torch_api.cpp:2:9: fatal error: 'torch/torch.h' file not found
#include<torch/torch.h>

i then tried a few variations of the LIBTORCH path, but none was able to let the torch.h header file be found...

[LaurentMazare]

Right, you may want to try playing withe various enviroment variables, note that when using LIBTORCH you should not put the include bit at the very end as it's added automatically for you.
You can have a look at the discover.ml file if you want to get a better sense of how it's used.

[nilsbecker]

fwiw, the homebrew installed .dylibs are in /usr/local/Cellar/pytorch/1.13.1/lib/. i'll have a look at discover.ml

[LaurentMazare]

I would try LIBTORCH=/usr/local/Cellar/pytorch/1.13.1 in this case, probably good if you look at the flags that it pass to the c++ compiler if it errors out to check that they point at the right directories.

[nilsbecker]

that appears to have done it! how can i install this into my switch now? make install? no i guess not. will opam do the right thing if i opam pin the directory after compilation?

[nilsbecker]

ok, i exported LIBTORCH and then opam pinned torch to the directory. opam reports dune build and this takes a while and reports no errors. so it appears it has indeed used the homebrew pytorch installation which is good news. in addition, the gpu version should also be available. so that's a good workaround

however, this will be a bit brittle as homebrew is bound to update the pytorch lib in the future, going out of sync with what ocaml-torch supports?

it would be better if without env variables, ocaml-torch could ignore the installed pytorches and instead use the vendored pytorch; this does work on a clean system. so i guess the original issue still stands. maybe discover.ml needs some adjustment? somehow doing the equivalent of setting LIBTORCH to the opam switch subdir that contains the vendored pytorch?

[nilsbecker]

addendum: in view of #77 i'm not sure why the build setting only LIBTORCH succeeded now. i think it's because i had copied the formerly missing libquadmath.0.dylib manually out of desperation. i currently have one in /usr/local/lib/libquadmath.0.dylib
so if anyone tries to follow this path, be sure to also make libquadmath available in some, ideally less brittle, way.
additionally, afair, i also had to copy one more dylib to the discoverable location: /usr/local/lib/libgcc_s.1.1.dylib

[nilsbecker]

I retried the procedure above on another 'fresh' intel mac, without a copy of libquadmath in /usr/local/lib.

• initial state: no pytorch installed
• brew install torch (1.13.1 gets installed)
• no local pin of ocaml-torch
• LIBTORCH=/usr/local/Cellar/pytorch/1.13.1 opam install torch

this fails with ld: file not found: @rpath/libquadmath.0.dylib

however, on the same machine, then doing the following succeeds:

• brew uninstall pytorch
• opam install torch

it seems in absence of homebrew provided pytorch, the absence of libquadmath.0.dylib is not a problem.

i then tried opam installation of a local pin. first, without installing homebrew provided pytorch

• opam pin add torch localtorchdir

this successfully builds and reinstalls ocaml-torch.

finally, local pin plus homebrew installed pytorch.

• brew install pytorch
• LIBTORCH=/usr/local/Cellar/pytorch/1.13.1 opam reinstall torch

this again fails with ld: file not found: @rpath/libquadmath.0.dylib for architecture x86_64

as on the other machine and on various answers on the internet i believe just copying the missing dylibs to /usr/local/lib would fix that, but it's fragile, wrong, poorly reproducible and ugly.

hth