djsutherland/cyflann

A Cython interface to FLANN

cyflann

cyflann is a Cython-based interface to the FLANN library. It allows you to call FLANN from Cython without python overheads; in particular, you can make many calls in parallel threads without the GIL.

If you're just looking for any Python interface to FLANN, the ctypes interface that it ships with may be better for you. I wrote this interface for an application that needs to run lots of independent searches without the GIL.

The interface is currently incomplete; right now only float32 is supported, it also has some known issues, and is probably less friendly in general than it could be. If you want to use it, bug reports and/or pull requests are welcome.

cyflann is only tested with FLANNs since 1.8.4. cyflann supports FLANN's OpenMP wrappers, but has not been tested with its MPI or CUDA interfaces.

Installation

If you use the Anaconda Python distribution, the easiest way to get both cyflann and FLANN, including on Windows, is:

conda install -c conda-forge cyflann

Otherwise, you need to install FLANN yourself (see below), and can then run:

pip install cyflann

NOTE: If you're using FLANN 1.8.4 or earlier, a problem with its pkg-config files means that cyflann won't link properly. This has been fixed in the development branch since April 2013, but the 1.9.1 release isn't yet widely packaged.

To work around this problem, set the environment variable FLANN_DIR to the root of the installation before running pip or setup.py, e.g. /usr/local if the libraries are in /usr/local/lib/libflann.so. If you're using sudo, remember that it doesn't necessarily propagate environment variables by default; sudo FLANN_DIR=/wherever pip install cyflann will work.

Installing FLANN

Anaconda: conda install -c conda-forge flann (included as a requirement by the cyflann package).

OSX: using Homebrew, brew install flann; this gives you 1.9.1, but if pkg-config doesn't work for you for some reason, it's FLANN_DIR=$(brew --prefix).

Ubuntu: apt-get install libflann1 flann-dev; set FLANN_DIR=/usr.

Fedora: yum install flann flann-devel; set FLANN_DIR=/usr.

CentOS: EPEL has flann packages, but they're old and not tested with cyflann. Compile from source.

Arch: Install the AUR flann package; cyflann wants FLANN_DIR=/usr.

From source: Download the release source, and follow the user manual to install. If you're installing 1.9.1 and have pkg-config available, you shouldn't need to set FLANN_DIR, but if not set it to whatever you set CMAKE_INSTALL_PREFIX to (/usr/local by default).

cyflann's Issues

nn_radius() returns no more than 32 neighbors

nn_radius only gives me 32 neighbors max. Despite max_nn=-1 or 1024 and also params.max_neighbors doesn't work?

support types other than float32

consider versiontools

http://versiontools.readthedocs.org/en/latest/

FLANN_DEFAULT_PARAMETERS linkage on Windows

For some reason, MSVC doesn't get FLANN_DEFAULT_PARAMETERS out of the import lib, even though it's there and objdump /exports lists it and everything.

https://ci.appveyor.com/project/dougalsutherland/cyflann/build/1.0.49

Would be nice to use this so we can stop redefining the default parameters and actually use them from the library.

publish location of flann for consumers to link against

So that e.g. py-sdm doesn't have to find flann itself, it just does library_dirs=[cyflann.get_library_dirs()] or whatever.

Add a test that FLANNExtension / etc works

Would help with making sure #18 makes sense.

Add a .pxd interface so other cython users can use it

build issue on OSX

(From djsutherland/py-sdm#32)

I am using anaconda and set up an environment as described. When I run:

 pip install 'https://github.com/dougalsutherland/py-sdm/tarball/master#egg=sdm-0.1.0dev'

I get the following error:

 ImportError: The Cython extension requires cyflann to be installed before compilation.

 Install cyflann (e.g. `pip install cyflann`), and then try again.

I just ran pip install cyflann and it still raises this exception.

Running python -c 'import cyflann' gives me another error:

ImportError: dlopen(/Users/wgmueller/anaconda/envs/py-sdm2/lib/python2.7/site-packages/cyflann/index.so, 2): Symbol not found: _DEFAULT_FLANN_PARAMETERS
Referenced from: /Users/wgmueller/anaconda/envs/py-sdm2/lib/python2.7/site-packages/cyflann/index.so
Expected in: dynamic lookup

Any suggestions?

Travis setup

Might be nice. Have it build and upload conda packages (and pip wheels?) too.

test error

$ python test_vs_pyflann.py 
Traceback (most recent call last):
  File "test_vs_pyflann.py", line 4, in <module>
    import cyflann
  File "/home/magnolia/cyflann/cyflann/__init__.py", line 1, in <module>
    from .index import FLANNParameters, FLANNIndex, set_distance_type
ImportError: /home/magnolia/cyflann/cyflann/index.so: undefined symbol: DEFAULT_FLANN_PARAMETERS

allow specifying nonstandard flann location in installation

Need to set include_dirs and extra_link_args appropriately.

Also, should publish these variables so consumers (like py-sdm) know where to compile/link to as well.

use the c++ api directly

Why not? It'll avoid a layer of indirection and maybe be simpler.

Probably conflicts with #6, but might make #9 easier.

Implement clustering functions

OSX: better support for nonstandard install paths

Passing an absolute path to the library name doesn't seem to be doing it for me anymore; not sure if this is a change in newer OSX versions or if I just didn't test appropriately earlier. Should manually run install_name_tool to fix it after compilation; example.

Handle random seeds appropriately

I currently only use the non-randomized functions so this doesn't matter, but still, should fix it.

doesn't compile with cython 0.20.*

Because of the struct/dict conversion issue discussed here and here.

Dunno of a workaround, other than (a) manually enumerating all the possible options or (b) disabling as_dict() / __repr__() (which uses it).

FLANNIndex handling of _data pointer

Have to figure out how to handle the _data pointer appropriately.

pickling not working properly on VC9

Don't know what's up with this.

https://ci.appveyor.com/project/dougalsutherland/cyflann/build/1.0.53/job/yknjkaaxgh0jt8f7

FLANNExtension shouldn't link to libflann_cpp when not necessary

Detect the language of the extension? Or just a keyword argument? Probably just a keyword argument.

compile flann with setup.py install?

Would it be possible to actually compile FLANN with this thing? So that pip install cyflann would work even if you didn't have flann installed. (Though you'd still need cmake, unless I wanted to really go crazy.)

Would probably need the option of linking to a system install of flann also, though.

is all the linking grossness still needed?

Anaconda uses rpath properly now. Was that the only reason we needed it?

Interface incompatability

It would probably be nice if cyflann were a drop-in replacement for pyflann, in case anyone other than py-sdm is ever going to want to use it.

Compile to a single library file ?

Hi Dougal,

Thanks for doing this. I'm pretty new to Cython and my C/C++ is from 20 years ago. I'd ideally like to wrap the FLANN C++ code in Cython and then compile it all to a single .so / .pyd file. Is this possible or will there always be two libs (one Cython, one FLANN native) ?

Thanks

Ian

Failed to install throuch pip or setup.py on OS X

Environment:

• OS X El Capitan 10.11.6
• Python: 2.7.11
• Numpy: 1.11.1
• Pip: 8.1.2
• FLANN: stable 1.8.4 (installed via Homebrew)

Could not install by either:

pip install cyflann

Or setup.py after checking out the code:

python setup.py install

Error message was:

...
cyflann/index.c:254:10: fatal error: 'numpy/arrayobject.h' file not found
#include "numpy/arrayobject.h"
         ^
1 error generated.
error: command 'clang' failed with exit status 1

Travis: test system installs of flann

Homebrew on OSX, apt on Ubuntu.

Implement nn(), nn_radius() functions