kyamagu / faiss-wheels Goto Github PK
View Code? Open in Web Editor NEWUnofficial faiss wheel builder
License: MIT License
Unofficial faiss wheel builder
License: MIT License
Github Actions support docker containers to easily build manylinux wheels.
Check how to install and build dependencies for faiss wheels.
Describe the bug
Doing a simple pip install of faiss-cpu and then importing the library into python will fail because it depends on numpy.
To Reproduce
Describe the steps to reproduce the behavior:
python3 -m pip install faiss-cpu
python3 "import faiss"
Sees the following error
Collecting faiss-cpu
Downloading faiss_cpu-1.7.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 24.8 MB/s eta 0:00:00
Installing collected packages: faiss-cpu
Successfully installed faiss-cpu-1.7.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Traceback (most recent call last):
File "/io/test-script.py", line 1, in <module>
import faiss
File "/usr/local/lib/python3.10/dist-packages/faiss/__init__.py", line 11, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
Expected behavior
Installing faiss-cpu should pull in all its dependencies automatically.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
import faiss
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\loader.py", line 34, in
from .swigfaiss import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\swigfaiss.py", line 13, in
from . import _swigfaiss
ImportError: DLL load failed while importing _swigfaiss: 找不到指定的模块。
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss_init_.py", line 17, in
from .loader import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\loader.py", line 39, in
from .swigfaiss import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\swigfaiss.py", line 13, in
from . import _swigfaiss
ImportError: DLL load failed while importing _swigfaiss: 找不到指定的模块。
Would you publish wheels for python3.8 ?
Describe the bug
Consistently getting the following error when installing the faiss-cpu==1.7.3 module on Mac OS (Silicon M2 Chip):
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
To Reproduce
pip install faiss-cpu==1.7.3
Expected behavior
When installing the package, it is expected that the package scripts should include a script called:
swigfaiss_avx2.py
But this is missing when trying to install the module on my mac and I think this is causing the issue
Desktop (please complete the following information):
Additional context
This happens when trying to "joblib.load()"
i install faiss-gpu by both pip install faiss-gpu and conda install faiss-gpu.
however, results is below:
Using cached faiss-gpu-1.6.3.tar.gz (17 kB)
Requirement already satisfied: numpy in d:\anaconda3\envs\deep_sort_pytorch\lib\site-packages (from faiss-gpu) (1.18.2)
Building wheels for collected packages: faiss-gpu
Building wheel for faiss-gpu (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: 'D:\anaconda3\envs\deep_sort_pytorch\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\ly\AppData\
Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py'"'"'; file='"'"'C:\Users\ly\AppData\Local\Temp\
pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().r
eplace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\ly\AppData\Local\Temp\pip-
wheel-3x2ioo54'
cwd: C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb
Complete output (36 lines):
running bdist_wheel
running build
running build_ext
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 111, in
ext_modules=[swigfaiss]
File "D:\anaconda3\envs\deep_sort_pytorch\lib\site-packages\setuptools_init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\site-packages\wheel\bdist_wheel.py", line 299, in run
self.run_command('build')
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 45, in run
build_ext.run(self)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 50, in build_extensions
self._remove_flag('-Wstrict-prototypes')
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
ImportError: DLL load failed: The specified module could not be found.
Can anyone help me with this? I've installed with pip install faiss-cpu and python version is 3.7.9
It's running good in colab, but not in local pc
I've installed library using pip install faiss-cpu, but while importing import faiss, it's returning error
Describe the bug
When running Windows 11 and Python 3.8.10 attempting to install the latest version of faiss-cpu I get AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
To Reproduce
pip install faiss-cpu
Expected behavior
faiss-cpu should be installed just fine.
Desktop (please complete the following information):
Additional context
I can resolve the problem by installing a previous version
pip install faiss-cpu==1.7.2
so it is not a blocker. I think I noticed on my other machine with Python 3.7 that faiss-cpu installed just fine without changing the version. I dont have it on hand to add any other details.
The CUDA 11.0 build in #56 bloats the wheel package size from 85.5 MB to 216.5 MB. Needs to investigate file size reduction.
Describe the bug
pip install faiss-cpu-noavx2
fails in a colab env:
!pip install faiss-cpu-noavx2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting faiss-cpu-noavx2
Using cached faiss-cpu-noavx2-1.6.3.tar.gz (16 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: faiss-cpu-noavx2
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for faiss-cpu-noavx2 (setup.py) ... error
ERROR: Failed building wheel for faiss-cpu-noavx2
Running setup.py clean for faiss-cpu-noavx2
Failed to build faiss-cpu-noavx2
To Reproduce
Describe the steps to reproduce the behavior:
just create a colab and execute the command pip install faiss-cpu-noavx2
Desktop (please complete the following information):
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Python version: 3.9.16
Additional context
I was trying to install https://pypi.org/project/image-embeddings/ when this error occured
Here is a colab if you want to reproduce: https://colab.research.google.com/drive/1yU5lyoRZ3xJbwf_UHldQtfWc-Tlaca3n?usp=sharing
When I install happens error
ERROR: Command errored out with exit status 1:
command: /home/sueoka/anaconda3/envs/nemo_v2/bin/python /home/sueoka/anaconda3/envs/nemo_v2/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmps_8f0m_q
cwd: /tmp/pip-install-kfbf4zqn/faiss-cpu_35de81a7d3fc4ea899e7130f7a81f594
Complete output (10 lines):
running bdist_wheel
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
swig error : Unrecognized option -doxygen
Use 'swig -help' for available options.
error: command '/usr/bin/swig' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu which use PEP 517 and cannot be installed directly
my env
SWIG Version 3.0.12
Compiled with g++ [powerpc64le-unknown-linux-gnu]
Configured options: +pcre
gcc (Ubuntu/IBM 5.5.0-12ubuntu1) 5.5.0 20171010 [ibm/gcc-5-branch revision 248295]
Python 3.8.0
Support was added to download the correct dependencies back in June, when will a new minor release be cut to incorporate this change?
code:
import numpy as np
import faiss
ver = np.random.randn(1000,512)
ver = ver .astype(np.float32)
print(ver .shape)
pq = faiss.ProductQuantizer(512, 4, 3) #dim:512 seg:4 bit:3
pq.train(ver)
codes = pq.compute_codes(ver)
print(codes.shape)
print(codes)
result: #got seg:2
(1000, 512)
(1000, 2)
[[ 58 5]
[ 62 7]
[ 63 11]
...
[171 7]
[ 57 7]
[ 92 1]]
hi, kyamagu,
Thanks for this repo!
Because I need py3.7, so I have to install faiss on py3.8
But without luck, here is the minial reproduce process:
docker run --rm --name Test-service -ti python:3.7-buster sh
// then in the container
# pip install faiss-cpu
WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting faiss-cpu
Downloading faiss_cpu-1.6.1-cp37-cp37m-manylinux2010_x86_64.whl (7.1 MB)
|████████████████████████████████| 7.1 MB 330 kB/s
Collecting numpy
Downloading numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
|████████████████████████████████| 20.1 MB 278 kB/s
Installing collected packages: numpy, faiss-cpu
Successfully installed faiss-cpu-1.6.1 numpy-1.18.1
# python -c "import faiss"
Illegal instruction (core dumped)
But python:3.8
seems fine. Have any suggestion? Thanks in advance!
I also search the
Illegal instruction (core dumped)
issues in facebook's faiss repo, but most of them are 1.5.3 problem.
Describe the bug
Can't build wheels.
To Reproduce
pip install faiss-cpu
Expected behavior
#11 827.7 swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
#11 827.7 swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -DSWIGWORDSIZE64 -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
#11 827.7 faiss/faiss/python/swigfaiss.i:268: Error: Unable to find 'faiss/impl/platform_macros.h'
#11 827.7 faiss/faiss/python/swigfaiss.i:272: Error: Unable to find 'faiss/utils/ordered_key_value.h'
#11 827.7 faiss/faiss/python/swigfaiss.i:273: Error: Unable to find 'faiss/utils/Heap.h'
#11 827.7 faiss/faiss/python/swigfaiss.i:278: Error: Unable to find 'faiss/utils/AlignedTable.h'
...
#11 827.7 [end of output]
#11 827.7
#11 827.7 note: This error originates from a subprocess, and is likely not a problem with pip.
#11 827.7 ERROR: Failed building wheel for faiss-cpu
#11 827.7 Successfully built numexpr numpy PyYAML
#11 827.7 Failed to build faiss-cpu
#11 827.7 ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects
Desktop (please complete the following information):
Hello! I am currently getting the following abort in my Python code:
OMP: Error #15: Initializing libomp.a, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous,
since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP
runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe,
unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow
the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information,
please see http://openmp.llvm.org/
[1] 24366 abort python train.py
I believe this is probably caused by a conflict between torch
and faiss-cpu
since my code relies on both (as well as numpy
but numpy is used by faiss). I'm trying to reproduce the issue outside of my code so I can post it here. I tried to compile a version myself by copying the build parameters used in the repository, however, I was unable to get the same performance so I gave up.
I am using faiss-cpu
version 1.7.0 on macOS 11.2.3
Describe the bug
Requested faiss-cpu from https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz has inconsistent name: expected 'faiss-gpu', but metadata has 'faiss-cpu'
To Reproduce
Describe the steps to reproduce the behavior:
pip install faiss-gpu
Desktop (please complete the following information):
Hi,
With faiss-cpu on linux I get an error:
Traceback:
env/lib/python3.6/site-packages/faiss/loader.py:31: in <module>
from .swigfaiss_avx2 import *
E ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
During handling of the above exception, another exception occurred:
env/lib/python3.6/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
xxx/tests/handlers/xxx.py:6: in <module>
from xxx.handlers import xxx as xxx_handler
xxx/handlers/xxx.py:2: in <module>
import faiss
env/lib/python3.6/site-packages/faiss/__init__.py:17: in <module>
from .loader import *
env/lib/python3.6/site-packages/faiss/loader.py:39: in <module>
from .swigfaiss import *
env/lib/python3.6/site-packages/faiss/swigfaiss.py:13: in <module>
from . import _swigfaiss
E ImportError: dlopen: cannot load any more object with static TLS
Seems related to other issues, but I'm still not sure of the solution:
facebookresearch/faiss#866
https://github.com/facebookresearch/faiss/issues?q=is%3Aissue+swigfaiss_avx2
Describe the bug
I am trying to install faiss-gpu.
To Reproduce
pip install faiss-gpu
outputs
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement faiss-gpu (from versions: none)
ERROR: No matching distribution found for faiss-gpu
I even tried to download the .whl file from https://pypi.org/project/faiss-gpu/#files
and input
pip install --use-wheel --no-index --find-links=[FILEPATH] faiss-gpu
Expected behavior
Normal installation like any other pip package
Desktop (please complete the following information):
Additional context
I am trying to download/install this, in order to make install
this https://github.com/Mangio621/Mangio-RVC-Fork
In the above's makefile
there is this line basically pip install faiss-gpu fairseq gradio ffmpeg ffmpeg-python praat-parselmouth pyworld numpy==1.23.5 numba==0.56.4 librosa==0.9.1
and it fails at the very first, others have no problem if i run them by themselves.
Hello, I am getting this error on windows 10 with python 3.9, faiss 1.7.2 after importing faiss. Same happens with conda-forge version.
Hey,
First of all: Thanks for maintaining the unofficial PyPI packages!
I noticed that with the latest 1.6.4. the installation via pip fails if you don't run the install from an env already having numpy installed. This is particularly problematic in fresh deployments / dockerfiles.
I see that you are using numpy here in setup.py. From my perspective, there should ideally be no external packages needed before the installation. Could you maybe move back to standard python modules there?
Line 31 in c11b8de
Error message:
Collecting faiss-cpu (from -r requirements.txt (line 19))
Downloading https://files.pythonhosted.org/packages/5d/3d/8ab01d4421c3889e8d3379661f5453ebbbec209f91ee4d067af13ca18a13/faiss-cpu-1.6.4.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-iiioulxg/faiss-cpu/setup.py", line 6, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-iiioulxg/faiss-cpu/
The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1
Hi, could you please update the python pip package with the changes in PR #2848 ?
facebookresearch/faiss#2843
facebookresearch/faiss#2848
facebookresearch/faiss#2988
Describe the bug
Running setup.py install for faiss-gpu did not run successfully
cuda_11.7
But CPU Versions works!
(env) PS C:\Server\htdocs\ml2> pip install faiss-gpu
Downloading faiss-gpu-1.6.4.post2.tar.gz (25 kB)
Preparing metadata (setup.py) ... done
WARNING: Generating metadata for package faiss-gpu produced metadata for project name faiss-cpu. Fix your #egg=faiss-gpu fragments.
Discarding https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz (from https://pypi.org/simple/faiss-gpu/): Requested faiss-cpu from https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz has inconsistent name: expected 'faiss-gpu', but metadata has 'faiss-cpu'
Downloading faiss-gpu-1.6.4.tar.gz (3.4 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
running egg_info
creating C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info
writing C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\PKG-INFO
writing dependency_links to C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\dependency_links.txt
writing top-level names to C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\top_level.txt
writing manifest file 'C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\SOURCES.txt'
error: package directory 'C:\Users\szcze\AppData\Local\Temp\pip-install-8jg13zcy\faiss-gpu_62ba4c34a62d4ef6981c53e51f8b7158\faiss\faiss\python' does not exist
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
To Reproduce
I am using Windows 10 and Python 3.8.10 (venv)
cuda_11.7.r11.7/compiler.31442593_0
(env) PS C:\Server\htdocs\ml2> pip install faiss-gpu
Expected behavior
A clear and concise description of what you expected to happen.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
For environment with faiss-cpu 1.7.4 and mkl 2023.1.0, python import faiss attempt to load libmkl_intel_lp64.so.1 while only libmkl_intel_lp64.so.2 is installed.
To Reproduce
Describe the steps to reproduce the behavior:
Installation is from anaconda and using conda channels: defaults, pytorch, conda-forge
conda config --add channels defaults
conda config --add channels pytorch
conda config --add channels conda-forge
conda update conda
conda install faiss-cpu==1.7.4 mkl=2023.1.0
import faiss
Expected behavior
Successful import
Actual behavior
>>> import faiss
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/miniconda/lib/python3.8/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/opt/miniconda/lib/python3.8/site-packages/faiss/loader.py", line 65, in <module>
from .swigfaiss import *
File "/opt/miniconda/lib/python3.8/site-packages/faiss/swigfaiss.py", line 13, in <module>
from . import _swigfaiss
ImportError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory
Desktop (please complete the following information):
Additional context
Discovered issue when use faiss-cpu and mkl under Azure Machine Learning environment.
Describe the bug
FAISS has different wheels for arm64 chips & Intel/AMD chips. If you vectorise a document using arm64, you should use FAISS to interact with this document on a arm64 machine.
So right now, my mac is arm64 and I vectorised docs on my mac using the arm64 wheel. However in the Azure functions, it attempts to use the Intel wheel which is slightly different. Intel/AMD chips use something called AVX2 support in FAISS (no idea what it actually is but it's different to the way arm64 machines use FAISS).
How can this be solved?
Normalise the wheels across all architectures to solve this compatibility issue problem - happy to help?
In faiss/Makefile
, the following is always true in CI environment regardless of CUDA installation. Check why.
ifneq ($(strip $(NVCC)),)
OBJ += $(GPU_OBJ)
HEADERS += $(GPU_HEADERS)
endif
Describe the bug
In FreeBSD pip install of faiss-cpu failed building wheel for faiss-cpu
To Reproduce
pip install faiss-cpu
Collecting faiss-cpu
Using cached faiss-cpu-1.7.4.tar.gz (57 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: faiss-cpu
Building wheel for faiss-cpu (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for faiss-cpu (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
running bdist_wheel
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
error: command 'swig' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects
Desktop:
The latest release of faiss-wheels is currently v1.7.4, but the latest release on PyPI is v1.7.2. This means that faiss-gpu
cannot be used with python 3.11 at the moment, even though support was already added in v1.7.3. Any chance we can update PyPI? Happy to help if there is work involved.
Faiss now supports windows conda build.
For wheels, there needs to be stable BLAS/LAPACK distribution that CMake can find. Perhaps OpenBLAS is the first option, but it is unknown how to resolve internal fortran-related dependency in msbuild environment.
Hey, It seems like the same issue as #38 prevails even in 1.7.1:
import faiss
C:\Python\lib\site-packages\faiss\__init__.py:18: in <module>
from .loader import *
C:\Python\lib\site-packages\faiss\loader.py:65: in <module>
from .swigfaiss import *
C:\Python\lib\site-packages\faiss\swigfaiss.py:13: in <module>
from . import _swigfaiss
E ImportError: DLL load failed while importing _swigfaiss: The specified module could not be found.
Windows 10, Python 3.9
Hi, I've been trying to follow the instructions to build faiss wheels with AVX2 and MKL.
I clone the repository, build faiss without python and GPU and run:
> export FAISS_OPT_LEVEL='avx2'
> export FAISS_LDFLAGS='-l:libfaiss_avx2.a -lgfortran'
> pip install --no-binary :all: faiss-cpu
Collecting faiss-cpu
Using cached faiss-cpu-1.6.5.tar.gz (28 kB)
Skipping wheel build for faiss-cpu, due to binaries being disabled for it.
Installing collected packages: faiss-cpu
Running setup.py install for faiss-cpu ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f04b57bf/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/faiss-cpu
cwd: /tmp/pip-install-ejk03m5r/faiss-cpu/
Complete output (60 lines):
running install
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
faiss/faiss/python/swigfaiss.i:229: Error: Unable to find 'faiss/impl/platform_macros.h'
faiss/faiss/python/swigfaiss.i:233: Error: Unable to find 'faiss/utils/ordered_key_value.h'
faiss/faiss/python/swigfaiss.i:234: Error: Unable to find 'faiss/utils/Heap.h'
faiss/faiss/python/swigfaiss.i:235: Error: Unable to find 'faiss/utils/partitioning.h'
faiss/faiss/python/swigfaiss.i:236: Error: Unable to find 'faiss/utils/hamming.h'
faiss/faiss/python/swigfaiss.i:339: Error: Unable to find 'faiss/utils/utils.h'
faiss/faiss/python/swigfaiss.i:340: Error: Unable to find 'faiss/utils/distances.h'
faiss/faiss/python/swigfaiss.i:341: Error: Unable to find 'faiss/utils/random.h'
faiss/faiss/python/swigfaiss.i:343: Error: Unable to find 'faiss/MetricType.h'
faiss/faiss/python/swigfaiss.i:344: Error: Unable to find 'faiss/Index.h'
faiss/faiss/python/swigfaiss.i:345: Error: Unable to find 'faiss/Clustering.h'
faiss/faiss/python/swigfaiss.i:347: Error: Unable to find 'faiss/utils/extra_distances.h'
faiss/faiss/python/swigfaiss.i:351: Error: Unable to find 'faiss/impl/ProductQuantizer.h'
faiss/faiss/python/swigfaiss.i:353: Error: Unable to find 'faiss/VectorTransform.h'
faiss/faiss/python/swigfaiss.i:354: Error: Unable to find 'faiss/IndexPreTransform.h'
faiss/faiss/python/swigfaiss.i:355: Error: Unable to find 'faiss/IndexFlat.h'
faiss/faiss/python/swigfaiss.i:356: Error: Unable to find 'faiss/IndexLSH.h'
faiss/faiss/python/swigfaiss.i:357: Error: Unable to find 'faiss/impl/PolysemousTraining.h'
faiss/faiss/python/swigfaiss.i:358: Error: Unable to find 'faiss/IndexPQ.h'
faiss/faiss/python/swigfaiss.i:359: Error: Unable to find 'faiss/InvertedLists.h'
faiss/faiss/python/swigfaiss.i:360: Error: Unable to find 'faiss/DirectMap.h'
faiss/faiss/python/swigfaiss.i:363: Error: Unable to find 'faiss/IndexIVF.h'
faiss/faiss/python/swigfaiss.i:368: Error: Unable to find 'faiss/IVFlib.h'
faiss/faiss/python/swigfaiss.i:369: Error: Unable to find 'faiss/impl/ScalarQuantizer.h'
faiss/faiss/python/swigfaiss.i:370: Error: Unable to find 'faiss/IndexScalarQuantizer.h'
faiss/faiss/python/swigfaiss.i:371: Error: Unable to find 'faiss/IndexIVFSpectralHash.h'
faiss/faiss/python/swigfaiss.i:372: Error: Unable to find 'faiss/impl/HNSW.h'
faiss/faiss/python/swigfaiss.i:373: Error: Unable to find 'faiss/IndexHNSW.h'
faiss/faiss/python/swigfaiss.i:374: Error: Unable to find 'faiss/IndexIVFFlat.h'
faiss/faiss/python/swigfaiss.i:379: Error: Unable to find 'faiss/OnDiskInvertedLists.h'
faiss/faiss/python/swigfaiss.i:382: Error: Unable to find 'faiss/impl/lattice_Zn.h'
faiss/faiss/python/swigfaiss.i:383: Error: Unable to find 'faiss/IndexLattice.h'
faiss/faiss/python/swigfaiss.i:386: Error: Unable to find 'faiss/IndexIVFPQ.h'
faiss/faiss/python/swigfaiss.i:387: Error: Unable to find 'faiss/IndexIVFPQR.h'
faiss/faiss/python/swigfaiss.i:388: Error: Unable to find 'faiss/Index2Layer.h'
faiss/faiss/python/swigfaiss.i:390: Error: Unable to find 'faiss/IndexBinary.h'
faiss/faiss/python/swigfaiss.i:391: Error: Unable to find 'faiss/IndexBinaryFlat.h'
faiss/faiss/python/swigfaiss.i:392: Error: Unable to find 'faiss/IndexBinaryIVF.h'
faiss/faiss/python/swigfaiss.i:393: Error: Unable to find 'faiss/IndexBinaryFromFloat.h'
faiss/faiss/python/swigfaiss.i:394: Error: Unable to find 'faiss/IndexBinaryHNSW.h'
faiss/faiss/python/swigfaiss.i:395: Error: Unable to find 'faiss/IndexBinaryHash.h'
faiss/faiss/python/swigfaiss.i:397: Error: Unable to find 'faiss/impl/ThreadedIndex.h'
faiss/faiss/python/swigfaiss.i:401: Error: Unable to find 'faiss/IndexShards.h'
faiss/faiss/python/swigfaiss.i:405: Error: Unable to find 'faiss/IndexReplicas.h'
faiss/faiss/python/swigfaiss.i:409: Error: Unable to find 'faiss/MetaIndexes.h'
faiss/faiss/python/swigfaiss.i:594: Error: Unable to find 'faiss/impl/io.h'
faiss/faiss/python/swigfaiss.i:595: Error: Unable to find 'faiss/index_io.h'
faiss/faiss/python/swigfaiss.i:596: Error: Unable to find 'faiss/clone_index.h'
faiss/faiss/python/swigfaiss.i:600: Error: Unable to find 'faiss/AutoTune.h'
faiss/faiss/python/swigfaiss.i:601: Error: Unable to find 'faiss/index_factory.h'
faiss/faiss/python/swigfaiss.i:602: Error: Unable to find 'faiss/MatrixStats.h'
faiss/faiss/python/swigfaiss.i:848: Error: Unable to find 'faiss/impl/AuxIndexStructures.h'
error: command 'swig' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f04b57bf/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/faiss-cpu Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
First of all, I notice that there is no swigfaiss.i
file — only swigfaiss.swig
. So I rename it and start building the local package instead of the pypi one:
> export FAISS_OPT_LEVEL='avx2'
> export FAISS_LDFLAGS='-l:libfaiss_avx2.a -lgfortran'
> python3 setup.py install
However, this fails because ldd cannot find libfaiss_avx2.a
. Then I specify it like so:
> export FAISS_LDFLAGS='-Lfaiss/build/faiss/libfaiss_avx2.a -lgfortran'
This results in a successful compilation. However, when I try to import faiss
:
> root@fb3557b7b9e5:/opt/instance/faiss/faiss_debug/faiss-wheels-compile/faiss-wheels2# ipython3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import faiss
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/loader.py in <module>
30 logger.info("Loading faiss with AVX2 support.")
---> 31 from .swigfaiss_avx2 import *
32 else:
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
<ipython-input-1-9e6a9f0b5bbf> in <module>
----> 1 import faiss
/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/__init__.py in <module>
15
16 # We import * so that the symbol foo can be accessed as faiss.foo.
---> 17 from .loader import *
18
19
/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/loader.py in <module>
37 # we import * so that the symbol X can be accessed as faiss.X
38 logger.info("Loading faiss.")
---> 39 from .swigfaiss import *
/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/swigfaiss.py in <module>
11 # Import the low-level C/C++ module
12 if __package__ or "." in __name__:
---> 13 from . import _swigfaiss
14 else:
15 import _swigfaiss
ImportError: /usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/_swigfaiss.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTIN5faiss9IndexFlatE
Okay, one more try. Beside building the faiss, I also install it using cmake --install build
. Then, again, I run:
> pip install --no-binary :all: faiss-spu
This also compiles successfully, but:
In [1]: import faiss
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/faiss/loader.py in <module>
30 logger.info("Loading faiss with AVX2 support.")
---> 31 from .swigfaiss_avx2 import *
32 else:
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
<ipython-input-1-9e6a9f0b5bbf> in <module>
----> 1 import faiss
/usr/local/lib/python3.8/dist-packages/faiss/__init__.py in <module>
15
16 # We import * so that the symbol foo can be accessed as faiss.foo.
---> 17 from .loader import *
18
19
/usr/local/lib/python3.8/dist-packages/faiss/loader.py in <module>
37 # we import * so that the symbol X can be accessed as faiss.X
38 logger.info("Loading faiss.")
---> 39 from .swigfaiss import *
/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py in <module>
11 # Import the low-level C/C++ module
12 if __package__ or "." in __name__:
---> 13 from . import _swigfaiss
14 else:
15 import _swigfaiss
ImportError: /usr/local/lib/python3.8/dist-packages/faiss/_swigfaiss.cpython-38-x86_64-linux-gnu.so: undefined symbol: dgemm_
Looks like the README instructions are not complete and the swigfaiss.swig
in the repo should be renamed. However, I still cannot build the wheel. Please advise
Running the faiss example
import numpy as np
d = 64 # dimension
nb = 100000 # database size
nq = 10000 # nb of queries
np.random.seed(1234) # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.
import faiss # make faiss available
index = faiss.IndexFlatL2(d) # build the index
print(index.is_trained)
index.add(xb) # add vectors to the index
print(index.ntotal)
k = 4 # we want to see 4 nearest neighbors
D, I = index.search(xb[:5], k) # sanity check
print(I)
print(D)
D, I = index.search(xq, k) # actual search
print(I[:5]) # neighbors of the 5 first queries
print(I[-5:]) # neighbors of the 5 last queries
I'm getting a segfault when doing the search.
OS: ubuntu 20.04
version : 1.6.3 of faiss-cpu
any idea what could be wrong ?
It would be really nice if this pip package could install faiss 1.6.3 official version.
I would have liked to make a PR for this but I am not sure how to proceed.
sdist
package does not bundle python_callbacks.h
required for building a wheel package.
There is an issue that the source tree has hard-coded #include <faiss/python/python_callbacks.h>
, that should be located inside the C++ header location. One option is to apply a patch to rewrite this to #include "python_callbacks.h"
and bundle the source in the source package.
Thanks for working on these wheels. They are very useful.
On Debian Stretch with Python 3.6, I get a segfault if I try to import faiss before tensorflow, but not vice-versa.
This should reproduce the issue:
docker run --rm -it python:3.6-stretch bash
pip install faiss-cpu tensorflow
python -c "import faiss; import tensorflow" # segfaults
python -c "import tensorflow; import faiss" # works
The segfault only occurs with faiss-cpu, it does not occur with faiss-gpu.
#33 seems to bring significant file size increase, causing pypi upload failure.
Needs to inspect the cause and find a workaround; perhaps by dropping some CUDA_ARCHITECTURE.
On aarch64, ‘pip install faiss-gpu’ builds the wheels from source code and gives the below error-
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjddlp5q/faiss-gpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjddlp5q/faiss-gpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-r241ub16
cwd: /tmp/pip-install-mjddlp5q/faiss-gpu/
Complete output (75 lines):
running bdist_wheel
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
faiss/faiss/python/swigfaiss.i:250: Error: Unable to find 'faiss/impl/platform_macros.h'
faiss/faiss/python/swigfaiss.i:254: Error: Unable to find 'faiss/utils/ordered_key_value.h'
faiss/faiss/python/swigfaiss.i:255: Error: Unable to find 'faiss/utils/Heap.h'
faiss/faiss/python/swigfaiss.i:260: Error: Unable to find 'faiss/utils/AlignedTable.h'
faiss/faiss/python/swigfaiss.i:261: Error: Unable to find 'faiss/utils/partitioning.h'
faiss/faiss/python/swigfaiss.i:262: Error: Unable to find 'faiss/utils/hamming.h'
faiss/faiss/python/swigfaiss.i:365: Error: Unable to find 'faiss/utils/utils.h'
faiss/faiss/python/swigfaiss.i:366: Error: Unable to find 'faiss/utils/distances.h'
faiss/faiss/python/swigfaiss.i:367: Error: Unable to find 'faiss/utils/random.h'
faiss/faiss/python/swigfaiss.i:369: Error: Unable to find 'faiss/MetricType.h'
faiss/faiss/python/swigfaiss.i:372: Error: Unable to find 'faiss/Index.h'
faiss/faiss/python/swigfaiss.i:373: Error: Unable to find 'faiss/Clustering.h'
faiss/faiss/python/swigfaiss.i:375: Error: Unable to find 'faiss/utils/extra_distances.h'
faiss/faiss/python/swigfaiss.i:379: Error: Unable to find 'faiss/impl/ProductQuantizer.h'
faiss/faiss/python/swigfaiss.i:380: Error: Unable to find 'faiss/impl/AdditiveQuantizer.h'
faiss/faiss/python/swigfaiss.i:381: Error: Unable to find 'faiss/impl/ResidualQuantizer.h'
On aarch64, ‘pip install faiss-gpu’ should download the wheels from PyPI
@kyamagu Please let me know your interest in releasing aarch64 wheels.
When I'm trying to install faiss-cpu with pip install --no-binary :all: faiss-cpu
pip installs 1.7.2 instead of 1.7.3, the latest one. Forcing installing faiss-cpu==1.7.3
shows that 1.7.2 is the latest one and there is no 1.7.3 at all.
The problem live with publishing to PyPi, because there is nothing in Source Distributions block on the PyPi page of the library, see here: https://pypi.org/project/faiss-cpu/#files, that's why pip can't build from the source.
I'm trying to follow https://github.com/kyamagu/faiss-wheels#linux so I can build my own binaries without AVX2
rom1504@rom1504-W35-37ET:~/faiss$ pip install --no-binary :all: faiss-cpu
Defaulting to user installation because normal site-packages is not writeable
Collecting faiss-cpu
Using cached faiss-cpu-1.6.3.tar.gz (17 kB)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from faiss-cpu) (1.16.2)
Skipping wheel build for faiss-cpu, due to binaries being disabled for it.
Installing collected packages: faiss-cpu
Running setup.py install for faiss-cpu ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ti_pc5up/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ti_pc5up/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sq3a3ca8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/rom1504/.local/include/python3.7m/faiss-cpu
cwd: /tmp/pip-install-ti_pc5up/faiss-cpu/
Complete output (56 lines):
running install
running build
running build_ext
building 'faiss._swigfaiss' extension
swigging python/swigfaiss.i to python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include/faiss -DSWIGWORDSIZE64 -o python/swigfaiss_wrap.cpp python/swigfaiss.i
python/swigfaiss.i:230: Error: Unable to find 'faiss/utils/Heap.h'
python/swigfaiss.i:231: Error: Unable to find 'faiss/utils/hamming.h'
python/swigfaiss.i:312: Error: Unable to find 'faiss/utils/utils.h'
These headers are actually present :
rom1504@rom1504-W35-37ET:~/faiss$ ls /usr/local/include/faiss
AutoTune.h Index2Layer.h IndexBinaryHNSW.h IndexHNSW.h IndexIVFPQR.h IndexPreTransform.h IVFlib.h utils
clone_index.h IndexBinaryFlat.h IndexBinaryIVF.h index_io.h IndexIVFSpectralHash.h IndexReplicas.h MatrixStats.h VectorTransform.h
Clustering.h IndexBinaryFromFloat.h index_factory.h IndexIVFFlat.h IndexLattice.h IndexScalarQuantizer.h MetaIndexes.h
DirectMap.h IndexBinary.h IndexFlat.h IndexIVF.h IndexLSH.h IndexShards.h MetricType.h
impl IndexBinaryHash.h Index.h IndexIVFPQ.h IndexPQ.h InvertedLists.h OnDiskInvertedLists.h
rom1504@rom1504-W35-37ET:~/faiss$ ls /usr/local/include/faiss/utils/
distances.h extra_distances.h hamming.h hamming-inl.h Heap.h random.h utils.h WorkerThread.h
What could be wrong ?
(cc @kyamagu for visibility)
There are currently no CPython 3.12 wheels on PyPI (https://pypi.org/project/faiss-cpu/#files). This causes an error on platforms using CPython 3.12 which are unable to build faiss-cpu
from source.
Can the contributors publish a new minor/patch release that includes CPython 3.12 for all platforms?
Example output, on an M1 Mac running CPython 3.12:
...
Building wheels for collected packages: faiss-cpu
Building wheel for faiss-cpu (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for faiss-cpu (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [12 lines of output]
running bdist_wheel
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
Traceback (most recent call last):
File "/opt/miniconda3/envs/jupyter-ai-jl4-pip/bin/swig", line 5, in <module>
from swig import swig
ModuleNotFoundError: No module named 'swig'
error: command '/opt/miniconda3/envs/jupyter-ai-jl4-pip/bin/swig' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects
hi,dear
any demo to test the faiss?
thx
use pip install faiss-cpu
got error, here is the log
Building wheels for collected packages: faiss-cpu
Building wheel for faiss-cpu (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\hx\AppData\Local\Temp\pip-wheel-3rrts8id'
cwd: C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\
Complete output (36 lines):
running bdist_wheel
running build
running build_ext
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 107, in <module>
ext_modules=[_swigfaiss]
File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
self.run_command('build')
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 45, in run
build_ext.run(self)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 50, in build_extensions
self._remove_flag('-Wstrict-prototypes')
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
----------------------------------------
ERROR: Failed building wheel for faiss-cpu
Running setup.py clean for faiss-cpu
Failed to build faiss-cpu
Installing collected packages: faiss-cpu
Running setup.py install for faiss-cpu ... error
ERROR: Command errored out with exit status 1:
command: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\黄 祥\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\hx\AppData\Local\Temp\pip-record-qsj3c0ie\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\ProgramData\Anaconda3\envs\blink37\Include\faiss-cpu'
cwd: C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\
Complete output (38 lines):
running install
running build
running build_ext
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 107, in <module>
ext_modules=[_swigfaiss]
File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\command\install.py", line 61, in run
return orig.install.run(self)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\install.py", line 545, in run
self.run_command('build')
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 45, in run
build_ext.run(self)
File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 50, in build_extensions
self._remove_flag('-Wstrict-prototypes')
File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\hx\AppData\Local\Temp\pip-record-qsj3c0ie\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\ProgramData\Anaconda3\envs\blink37\Include\faiss-cpu' Check the logs for full command output.
It seems the following fix enables faiss build on ARM64 mac. Good to investigate further packaging issues.
facebookresearch/faiss#1655
Describe the bug
faiss may crash when there is another imported package that depends on libomp
.
To Reproduce
Import a package that depends on OpenMP on macOS, then import faiss-cpu
package. Use faiss as usual.
Expected behavior
Should not segfault.
Desktop (please complete the following information):
Additional context
facebookresearch/faiss#2463
benfred/implicit#613 (comment)
dmlc/xgboost#7039 (comment)
https://bugs.llvm.org/show_bug.cgi?id=50579
When I install faiss-gpu via pip, index_cpu_to_gpu() seems to hang forever. For example, this code sample hangs for me:
import faiss
index_flat = faiss.IndexFlatL2(16)
gpu_index_flat = faiss.index_cpu_to_gpu(faiss.StandardGpuResources(), 0, index_flat)
The index_cpu_to_gpu() will hang, spinning a CPU at 100% seemingly forever, mostly in libnvidia-ptxjitcompiler.so.510.54
.
I confirmed this on two different machines with different GPUs (A100 and A10G).
I found two workarounds:
Maybe faiss-gpu wheel isn't built with CUDA 11 support, and that's why it doesn't work with A100 / A10G?
ERROR: Failed building wheel for faiss-gpu
hi,dear
when I install the faiss-cpu with
pip install faiss-cpu
then I got the error
Requirement already satisfied: numpy in d:\python\lib\site-packages (from faiss-cpu) (1.16.4)
Building wheels for collected packages: faiss-cpu
Running setup.py bdist_wheel for faiss-cpu: started
Running setup.py bdist_wheel for faiss-cpu: finished with status 'error'
Complete output from command d:\python\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\xulm1\\AppData\\Local\\Temp\\pip-install-o6e2hn4b\\faiss-cpu\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\xulm1\AppData\Local\Temp\pip-wheel-8av9rpam --python-tag cp36:
running bdist_wheel
running build
running build_ext
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 107, in <module>
ext_modules=[_swigfaiss]
File "d:\python\lib\site-packages\setuptools\__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "d:\python\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "d:\python\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "d:\python\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "d:\python\lib\site-packages\wheel\bdist_wheel.py", line 192, in run
self.run_command('build')
File "d:\python\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "d:\python\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "d:\python\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "d:\python\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "d:\python\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 45, in run
build_ext.run(self)
File "d:\python\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 50, in build_extensions
self._remove_flag('-Wstrict-prototypes')
File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
So could you pls help me?
thx
Describe the bug
Get a CUDA error when searching in a Faiss index having too small vectors
Faiss assertion 'err__ == cudaSuccess' failed in void faiss::gpu::ivfInterleavedScanImpl_32_(faiss::gpu::Tensor<float, 2, true>&, faiss::gpu::Tensor<long int, 2, true>&, faiss::gpu::DeviceVector<void*>&, faiss::gpu::DeviceVector<void*>&, faiss::gpu::IndicesOptions, faiss::gpu::DeviceVector<int>&, int, faiss::MetricType, bool, faiss::gpu::Tensor<float, 3, true>&, faiss::gpu::GpuScalarQuantizer*, faiss::gpu::Tensor<float, 2, true>&, faiss::gpu::Tensor<long int, 2, true>&, faiss::gpu::GpuResources*) at /project/faiss/faiss/gpu/impl/scan/IVFInterleaved32.cu:13; details: CUDA error 9 invalid configuration argument
Aborted (core dumped)
or
Faiss assertion 'err__ == cudaSuccess' failed in int faiss::gpu::getNumDevices() at /project/faiss/faiss/gpu/utils/DeviceUtils.cu:36; details: CUDA error 401 the operation cannot be performed in the present state
To Reproduce
import faiss
import numpy as np
nb_vectors = 80_000
dim = 128
k = 5
nlist = 10
nprobe = 2
vectors = np.random.rand(nb_vectors, dim)
index = faiss.index_factory(dim, "IVF16384,Flat")
index.nlist = nlist
index.nprobe = nprobe
options = faiss.GpuMultipleClonerOptions()
options.shard = True
options.common_ivf_quantizer = True
index = faiss.index_cpu_to_all_gpus(index, options)
index.train(vectors)
index.add(vectors)
results = index.search(vectors, k)
This code works with dim = 1024
.
I also tried with different indexes and different parameters (nlist, etc.) but it always fails for a certain vector size (and not when increasing the size).
When I try to install different versions of Faiss (nightly and old version) I face incompatibility issues such as:
AttributeError: module 'faiss._swigfaiss' has no attribute 'delete_ParameterRangeVector'
or
TypeError: in method 'GpuIndexIVFFlat_train', argument 3 of type 'float const *'
Desktop:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.