jakevdp/pycrfsuite

pyCRFsuite

This is a python wrapper for crfsuite, a fast implementation of Conditional Random Fields

Authors

• Jake Vanderplas <[email protected]>

Installation

Currently the package is set-up only for in-place installation. It requires the crfsuite library to be installed: see http://www.chokkan.org/software/crfsuite/

Once this is installed, simply type make in the head directory.

Testing

There are a few basic test scripts in the head directory. test.py will read a small dataset from example_files, then run a basic training and tagging operation. crfsuite_test.sh runs the same operation using the command-line frontend provided by crfsuite. To compare the results of the training and tagging, run compare_output.sh. This will print all the places where the tagging results differ.

TODO

This is still a very incomplete wrapper. Search TODO within src/crfsuite.pyx to see some issues that need to be addressed.

Issues

There are a few 'features' in crfsuite that make efficient python wrapping difficult.

• Model File Output: as currently written, crfsuite writes the result of a training directly to a binary file. The library is not configured to allow writing the model to memory. This means that a python wrapper must write the model to disk, then read the model into memory before performing any tagging operation. It would be better if the model could be saved directly to a CRFsuite model structure, though when dealing with the very large datasets for which crfsuite is designed, it's clear why the author made the choice he did.
• Memory mapping: as currently written, crfsuite data is not stored in contiguous arrays. This means that there is no way to map a crfsuite data structure to a numpy array, and any input to crfsuite will need to be copied in memory. Addressing this would require significant upstream changes: the crfsuite_item_t structure would have to use an array of floats and an array of ints rather than an array of attribute structures.

pycrfsuite's Issues

Fixes branch open

Hi,

I've opened a branch called fixes in my fork where I'll push minor stuff. I'm not opening pull requests for trivialities, so you might want to check it periodically and merge/rebase what you think is worthwhile.

Regards,
Lars

Strange output values

I am trying to understand the output array from tagger. The console output seems correct, but if I check the numpy array I have strange results. For the first instance:
[ 8589934592 2 8589934594 3 3 8589934594
1 4294967298 8589934592 17179869187 8589934592 1
8589934594 25769803778 12884901888 8589934592 4294967299 12884901888
4 8589934594 4294967298 8589934592 6 25769803778
1 25769803778 8589934592 8589934594]

Am I having some kind of strange casting error ?

Issues in making

Hi, after installing crfsuite, cloned this repo and tried to 'make'.

First encountered the following:

user@computer:~/stuff/repos/pyCRFsuite$ make
python setup.py build_ext --inplace
running build_ext
skipping 'src/crfsuite.c' Cython extension (up-to-date)
building 'crfsuite' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Iinclude -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c src/crfsuite.c -o build/temp.linux-x86_64-2.7/src/crfsuite.o
src/crfsuite.c:225:22: fatal error: crfsuite.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:4: recipe for target 'crfsuite.so' failed
make: *** [crfsuite.so] Error 1

Noting that crfsuite had created a crfsuite.h elsewhere, I copied that file into the src directory and tried to 'make' again. This time, I got:

user@computer:~/stuff/repos/pyCRFsuite$ make
python setup.py build_ext --inplace
running build_ext
skipping 'src/crfsuite.c' Cython extension (up-to-date)
building 'crfsuite' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Iinclude -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c src/crfsuite.c -o build/temp.linux-x86_64-2.7/src/crfsuite.o
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1777:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/crfsuite.c:228:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
src/crfsuite.c: In function ‘__pyx_f_8crfsuite_progress’:
src/crfsuite.c:1429:30: error: format not a string literal and no format arguments [-Werror=format-security]
         fprintf(__pyx_v_fpo, __pyx_k_2);
                              ^
src/crfsuite.c: In function ‘__pyx_f_8crfsuite_read_data’:
src/crfsuite.c:1629:24: error: format not a string literal and no format arguments [-Werror=format-security]
   fprintf(__pyx_v_fpo, __pyx_k__0);
                        ^
src/crfsuite.c:1675:19: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     __pyx_v_token = iwa_read(__pyx_v_iwa);
                   ^
src/crfsuite.c:1978:24: error: format not a string literal and no format arguments [-Werror=format-security]
   fprintf(__pyx_v_fpo, __pyx_k_3);
                        ^
src/crfsuite.c: In function ‘__pyx_f_8crfsuite_read_data_for_tagging’:
src/crfsuite.c:2178:24: error: format not a string literal and no format arguments [-Werror=format-security]
   fprintf(__pyx_v_fpo, __pyx_k__0);
                        ^
src/crfsuite.c:2224:19: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     __pyx_v_token = iwa_read(__pyx_v_iwa);
                   ^
src/crfsuite.c:2566:24: error: format not a string literal and no format arguments [-Werror=format-security]
   fprintf(__pyx_v_fpo, __pyx_k_3);
                        ^
src/crfsuite.c: In function ‘__pyx_f_8crfsuite_output_tagging_result’:
src/crfsuite.c:2805:84: warning: passing argument 3 of ‘__pyx_v_labels->to_string’ from incompatible pointer type [-Wincompatible-pointer-types]
       __pyx_v_labels->to_string(__pyx_v_labels, (__pyx_v_inst->labels[__pyx_v_i]), (&__pyx_v_label));
                                                                                    ^
src/crfsuite.c:2805:84: note: expected ‘const char **’ but argument is of type ‘char **’
src/crfsuite.c:2835:76: warning: passing argument 3 of ‘__pyx_v_labels->to_string’ from incompatible pointer type [-Wincompatible-pointer-types]
     __pyx_v_labels->to_string(__pyx_v_labels, (__pyx_v_output[__pyx_v_i]), (&__pyx_v_label));
                                                                            ^
src/crfsuite.c:2835:76: note: expected ‘const char **’ but argument is of type ‘char **’
src/crfsuite.c:2892:26: error: format not a string literal and no format arguments [-Werror=format-security]
     fprintf(__pyx_v_fpo, __pyx_k_3);
                          ^
src/crfsuite.c:2902:24: error: format not a string literal and no format arguments [-Werror=format-security]
   fprintf(__pyx_v_fpo, __pyx_k_3);
                        ^
src/crfsuite.c: In function ‘__pyx_pf_8crfsuite_7CRFDict_7get_key’:
src/crfsuite.c:3563:95: warning: passing argument 3 of ‘((struct __pyx_obj_8crfsuite_CRFDict *)__pyx_v_self)->dict->to_string’ from incompatible pointer type [-Wincompatible-pointer-types]
   ((struct __pyx_obj_8crfsuite_CRFDict *)__pyx_v_self)->dict->to_string(__pyx_t_1, __pyx_t_2, (&__pyx_v_c));
                                                                                               ^
src/crfsuite.c:3563:95: note: expected ‘const char **’ but argument is of type ‘char **’
src/crfsuite.c: In function ‘__pyx_pf_8crfsuite_7CRFDict_8get_key_list’:
src/crfsuite.c:3725:146: warning: passing argument 3 of ‘((struct __pyx_obj_8crfsuite_CRFDict *)__pyx_v_self)->dict->to_string’ from incompatible pointer type [-Wincompatible-pointer-types]
     ((struct __pyx_obj_8crfsuite_CRFDict *)__pyx_v_self)->dict->to_string(((struct __pyx_obj_8crfsuite_CRFDict *)__pyx_v_self)->dict, __pyx_v_i, (&__pyx_v_c));
                                                                                                                                                  ^
src/crfsuite.c:3725:146: note: expected ‘const char **’ but argument is of type ‘char **’
src/crfsuite.c: In function ‘__pyx_pf_8crfsuite_10CRFDataset___init__’:
src/crfsuite.c:3820:13: warning: variable ‘__pyx_v_labels’ set but not used [-Wunused-but-set-variable]
   PyObject *__pyx_v_labels = 0;
             ^
src/crfsuite.c:3819:13: warning: variable ‘__pyx_v_attrs’ set but not used [-Wunused-but-set-variable]
   PyObject *__pyx_v_attrs = 0;
             ^
src/crfsuite.c: In function ‘__pyx_pf_8crfsuite_10CRFTrainer___init__’:
src/crfsuite.c:8169:164: warning: passing argument 3 of ‘((struct __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer->set_message_callback’ from incompatible pointer type [-Wincompatible-pointer-types]
  __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer->set_message_callback(((struct __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer, NULL, (&__pyx_f_8
                                                                                                                                                        ^
src/crfsuite.c:8169:164: note: expected ‘crfsuite_logging_callback {aka int (*)(void *, const char *, __va_list_tag *)}’ but argument is of type ‘int (*)(void *, char *, __va_list_tag *)’
src/crfsuite.c:8181:164: warning: passing argument 3 of ‘((struct __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer->set_message_callback’ from incompatible pointer type [-Wincompatible-pointer-types]
  __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer->set_message_callback(((struct __pyx_obj_8crfsuite_CRFTrainer *)__pyx_v_self)->trainer, NULL, (&__pyx_f_8
                                                                                                                                                        ^
src/crfsuite.c:8181:164: note: expected ‘crfsuite_logging_callback {aka int (*)(void *, const char *, __va_list_tag *)}’ but argument is of type ‘int (*)(void *, char *, __va_list_tag *)’
src/crfsuite.c: In function ‘__pyx_pf_8crfsuite_crfsuite_learn’:
src/crfsuite.c:10209:13: warning: variable ‘__pyx_v_rseed’ set but not used [-Wunused-but-set-variable]
   PyObject *__pyx_v_rseed = 0;
             ^
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:27:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/crfsuite.c:228:
src/crfsuite.c: At top level:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1448:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
 _import_array(void)
 ^
cc1: some warnings being treated as errors
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:4: recipe for target 'crfsuite.so' failed
make: *** [crfsuite.so] Error 1

Not sure what I'm doing wrong, and maybe this isn't the way to highlight this. Sorry, I'm new to this, but can you help?

features with pyCRFsuite

Hi ! I work with CRF using pyCRFsuite library and in my case I must use item sequences of float as features. I want to know more about the type of input features are there continuous or discrete values ?