Error in conda package installation

[facebookresearch]

Error in conda package installation

[bottler]

I think the InvalidVersionSpecError comes from using an old version of conda itself. I am using the 2020.11 anaconda module on devfair, which is conda 4.9.2, and which I think is the recommended one.

[pclucas14]

Ok after working with a more recent conda it worked (with python =3.7). Thanks for the support and closing the issue

[pclucas14]

ok willdo, thanks Priya!

[pclucas14]

OK let me try with a better conda version then

[prigoyal]

@pclucas14 , just an update here: @bottler uploaded new v0.1.5 conda packages that should now also work with python=3.8.
You should still have the conda 4.9.2 or latest conda otherwise all good to go! :)

[JinYeJin]

I updated conda version with just conda update conda, and it worked for me.

[pclucas14]

I also tried on another server, and it's still failing on the same step

(vissl) lpagec@dp-gpu1[pytorch]$conda install -c vissl -c iopath -c conda-forge -c pytorch -c defaults apex vissl
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining conflict for olefile ca-certificates typing_extensions mkl-service Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy ninja six Examining conflict for olefile typing_extensions mkl-service numpy freetype nExamining conflict forExamining conflict for olefile typing_extensions mkl-service numpy freetype ninja six libstdcxx-ng nump-

[962086838]

Same step and same issue on my Ubuntu 18.04 server.

[prigoyal]

thank you for reporting this @pclucas14 and @962086838. I am able to reproduce this too myself now which is strange as we haven't updated VISSL packages recently the uploaded packages worked seamlessly few days ago. If you are immediately blocked, you can install vissl in conda from source https://github.com/facebookresearch/vissl/blob/master/INSTALL.md#Install-from-source-in-Conda-environment which works semalessly

Assigning to @bottler to take a look at our conda packages. It seems like apex installs fine and something is not right with the vissl package only.

Looking a bit to identify what package is causing this issue:

(vissl-test2) prigoyal@devfair0209:~/.conda/envs/vissl-test2/lib/python3.8/site-packages$ grep -r "=2.7"
wheel-0.36.2-py3.6.egg-info/PKG-INFO:Requires-Python: !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7
setuptools/ssl_support.py:            # http://bugs.python.org/issue7776: Python>=3.4.1 and >=2.7.7
olefile-0.46.dist-info/METADATA:Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
six-1.15.0.dist-info/METADATA:Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*
mkl_service-2.3.0-py3.8.egg-info/PKG-INFO:Requires-Python: >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*
mkl_random-1.2.0-py3.8.egg-info/PKG-INFO:Requires-Python: >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*
numpy-1.19.2.dist-info/RECORD:numpy/random/_sfc64.cpython-38-x86_64-linux-gnu.so,sha256=2L7eIJmqQaR5oMpQc0Y6fJxU9UNMznLXIIVWuWey8N8,75600
caffe2/python/test/executor_test_util.py:        image_size=227,
Binary file caffe2/python/caffe2_pybind11_state_gpu.cpython-38-x86_64-linux-gnu.so matches
Binary file torch/bin/protoc matches
Binary file torch/bin/protoc-3.11.4.0 matches
Binary file torch/lib/libtorch_cpu.so matches
Binary file torch/lib/libtorch_cuda.so matches
Binary file torchvision/_C.so matches
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``1.4.0``                | ``0.5.0``                | ``==2.7``, ``>=3.5``, ``<=3.8`` |
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``1.3.1``                | ``0.4.2``                | ``==2.7``, ``>=3.5``, ``<=3.7`` |
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``1.3.0``                | ``0.4.1``                | ``==2.7``, ``>=3.5``, ``<=3.7`` |
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``1.2.0``                | ``0.4.0``                | ``==2.7``, ``>=3.5``, ``<=3.7`` |
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``1.1.0``                | ``0.3.0``                | ``==2.7``, ``>=3.5``, ``<=3.7`` |
torchvision-0.8.2-py3.8.egg-info/PKG-INFO:        | ``<=1.0.1``              | ``0.2.2``                | ``==2.7``, ``>=3.5``, ``<=3.7`` |

seems like the error is coming from torchvision which is strange as well.

[bottler]

This is an opencv weirdness.

First observation: The command works fine if I replace python=3.8 with python=3.7. The version of python shouldn't matter much. This is helpful, because we have a baseline. (If you don't actually need Python3.8 and are just trying to use vissl, the rest of this comment can be ignored!)

There are several packages which exist in more than one place on anaconda cloud. I have separated out all the dependencies. The following goes straight through, with the last line only pulling in apex and vissl. When I say "straight through", I mean the conda solver doesn't seem to need to work hard.

conda create -p /raid/reizenstein/jeremyconda/v4_p37 python=3.7 scikit-learn scipy parameterized
conda activate /raid/reizenstein/jeremyconda/v4_p37
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2
conda install -c conda-forge opencv hydra-core pycocotools tensorboard faiss 
conda install -c iopath -c fvcore -c conda-forge fvcore
conda install -v -c vissl -c fvcore -c iopath -c conda-forge -c pytorch -c defaults vissl=0.1.5

There are some background things to be aware of, but none are the primary problem here:

• The recommended way to install faiss is with faiss-gpu from the pytorch channel, but we depend on faiss which is in conda-forge. They have different dependencies, but the conda-forge one depends on pytorch and we need to avoid pulling in a pytorch from a default channel, which would be an unofficial build. Hence installing pytorch before any conda-forge packages. Ideally we would switch to depend on faiss-gpu. After that change, I would hope that the first two conda install lines could be swapped.
• There is a tensorboard package on both the anaconda and conda-forge channels and they have different dependencies.
• I have an initial build of vissl (0.1.0) sitting on anaconda cloud. This is from before we decided to put the classy vision code inside the vissl package instead of its own package. I specify the version of vissl in the above to avoid touching the old one. When the next release happens we should delete all the previous ones.
• The three packages scikit-learn, scipy, and parameterized exist both in anaconda and conda-forge and it doesn't seem to matter which ones you use.

When I try the same recipe with python 3.8, it chokes at the conda-forge stage. Just conda install -c conda-forge opencv alone is problematic for conda. The opencv package for Python 3.8 on conda-forge seems to be quite unfriendly to install. If you start an environment with conda create -p /raid/reizenstein/jeremyconda/foo python=3.8 and immediately try conda install -c conda-forge opencv, it works but insists on a downgrade from Python 3.8.8 to Python 3.8.3. If I try conda install -c defaults -c conda-forge opencv it works more smoothly. There may be a way forward here to make things work.

Basically opencv is a bit "special". It would be better to not explicitly depend on opencv, and tell the user to install it via pip. Similar to how our pip packages work. Or even, could we use PIL for the functionality we currently use in opencv, and therefore not need it at all.

[prigoyal]

thank you @bottler ! It seems like our immediate fix is to just recommend python=3.7 in the conda installation to the user. This is probably something I should update the website and our install instructions immediately.

For our next release, I agree with all the suggestions. I think opencv is a dependency but we can make it optional as we only have 0.1% of the code that requires opencv and otherwise we use PIL.
For the faiss, we should probably just switch to faiss-gpu but again , this is also a dependency that can become optional. I'll add a follow-up for both of these tasks!

[pclucas14]

Hi,

I tried the following commands with python 3.7

conda create -n vissl2 python=3.7
conda activate vissl2
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2
conda install -c vissl -c iopath -c conda-forge -c pytorch -c defaults apex vissl

And still I'm getting

(vissl2) lucaspc@devfair0224:~/repos/swav$ conda install -c vissl -c iopath -c conda-forge -c pytorch -c defaults apex vissl
Solving environment: failed


InvalidVersionSpecError: Invalid version spec: =2.7

[prigoyal]

thank you @pclucas14 , we are working on building new packages with the dependencies reduced! In the meantime, if you are blocked, I recommend 2 options:

1. If conda is must, build vissl from scratch following the instructions https://github.com/facebookresearch/vissl/blob/master/INSTALL.md#Install-from-source-in-Conda-environment
2. Use PIP packages of VISSL instead