bnosac/crfsuite

Labelling Sequential Data in Natural Language Processing with R - using CRFsuite

crfsuite's Issues

keep colnames of features in the model object

Vignette coefficients extraction returns NULL

When going through the vignette example of how to use crfsuite (which is very insightful, so thank you), the weights <- coefficients(model, encoding = "UTF-8") command returns NULL.

This is the same for "conll2002-nl" and "wikiner-en-wp3". Using coef instead of coefficients doesn't change this.

Are the weights/coefs the same as the state_features downloaded to summary(model, "modeldetails.txt")?

This is using crfsuite version 0.4 and running on R4.1.0 on Mac Monterey.

Thanks.

crf/src/crf1d_model.c:127:12: warning: writing 16 bytes into a region of size 4 [-Wstringop-overflow=]

Hi,
Thanks for using data.table. In revdep testing for the next release, I'm seeing the following compiler warning for crfsuite. I see everything is fully OK on CRAN, though. I have just upgraded to gcc 10.2.0 so maybe that's why. I'm surprised though as usually CRAN is the first to use latest compilers and spot things like this first.
Best, Matt

* installing *source* package ‘crfsuite’ ...
** package ‘crfsuite’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c cqdb/src/cqdb.c -o cqdb/src/cqdb.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c cqdb/src/lookup3.c -o cqdb/src/lookup3.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c liblbfgs/lib/lbfgs.c -o liblbfgs/lib/lbfgs.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_context.c -o crf/src/crf1d_context.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_encode.c -o crf/src/crf1d_encode.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_feature.c -o crf/src/crf1d_feature.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_model.c -o crf/src/crf1d_model.o
In function ‘read_uint8_array’,
    inlined from ‘crf1dm_new_impl’ at crf/src/crf1d_model.c:735:10:
crf/src/crf1d_model.c:127:12: warning: writing 16 bytes into a region of size 4 [-Wstringop-overflow=]
  127 |     *value = *buffer;
      |     ~~~~~~~^~~~~~~~~
crf/src/crf1d_model.c: In function ‘crf1dm_new_impl’:
crf/src/crf1d_model.c:64:17: note: at offset 0 to object ‘magic’ with size 4 declared here
   64 |     uint8_t     magic[4];       /* File magic. */
      |                 ^~~~~
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_tag.c -o crf/src/crf1d_tag.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crfsuite.c -o crf/src/crfsuite.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crfsuite_train.c -o crf/src/crfsuite_train.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/dataset.c -o crf/src/dataset.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/dictionary.c -o crf/src/dictionary.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/holdout.c -o crf/src/holdout.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/logging.c -o crf/src/logging.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/params.c -o crf/src/params.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/quark.c -o crf/src/quark.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/rumavl.c -o crf/src/rumavl.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_arow.c -o crf/src/train_arow.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_averaged_perceptron.c -o crf/src/train_averaged_perceptron.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_l2sgd.c -o crf/src/train_l2sgd.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_lbfgs.c -o crf/src/train_lbfgs.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_passive_aggressive.c -o crf/src/train_passive_aggressive.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-8T8CYO/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c rcpp_crfsuite.cpp -o rcpp_crfsuite.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-8T8CYO/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o crfsuite.so cqdb/src/cqdb.o cqdb/src/lookup3.o liblbfgs/lib/lbfgs.o crf/src/crf1d_context.o crf/src/crf1d_encode.o crf/src/crf1d_feature.o crf/src/crf1d_model.o crf/src/crf1d_tag.o crf/src/crfsuite.o crf/src/crfsuite_train.o crf/src/dataset.o crf/src/dictionary.o crf/src/holdout.o crf/src/logging.o crf/src/params.o crf/src/quark.o crf/src/rumavl.o crf/src/train_arow.o crf/src/train_averaged_perceptron.o crf/src/train_l2sgd.o crf/src/train_lbfgs.o crf/src/train_passive_aggressive.o rcpp_crfsuite.o RcppExports.o -L/usr/lib/R/lib -lR
rm -f cqdb/src/cqdb.o cqdb/src/lookup3.o liblbfgs/lib/lbfgs.o crf/src/crf1d_context.o crf/src/crf1d_encode.o crf/src/crf1d_feature.o crf/src/crf1d_model.o crf/src/crf1d_tag.o crf/src/crfsuite.o crf/src/crfsuite_train.o crf/src/dataset.o crf/src/dictionary.o crf/src/holdout.o crf/src/logging.o crf/src/params.o crf/src/quark.o crf/src/rumavl.o crf/src/train_arow.o crf/src/train_averaged_perceptron.o crf/src/train_l2sgd.o crf/src/train_lbfgs.o crf/src/train_passive_aggressive.o rcpp_crfsuite.o RcppExports.o 
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c cqdb/src/cqdb.c -o cqdb/src/cqdb.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c cqdb/src/lookup3.c -o cqdb/src/lookup3.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c liblbfgs/lib/lbfgs.c -o liblbfgs/lib/lbfgs.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_context.c -o crf/src/crf1d_context.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_encode.c -o crf/src/crf1d_encode.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_feature.c -o crf/src/crf1d_feature.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_model.c -o crf/src/crf1d_model.o
In function ‘read_uint8_array’,
    inlined from ‘crf1dm_new_impl’ at crf/src/crf1d_model.c:735:10:
crf/src/crf1d_model.c:127:12: warning: writing 16 bytes into a region of size 4 [-Wstringop-overflow=]
  127 |     *value = *buffer;
      |     ~~~~~~~^~~~~~~~~
crf/src/crf1d_model.c: In function ‘crf1dm_new_impl’:
crf/src/crf1d_model.c:64:17: note: at offset 0 to object ‘magic’ with size 4 declared here
   64 |     uint8_t     magic[4];       /* File magic. */
      |                 ^~~~~
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crf1d_tag.c -o crf/src/crf1d_tag.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crfsuite.c -o crf/src/crfsuite.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/crfsuite_train.c -o crf/src/crfsuite_train.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/dataset.c -o crf/src/dataset.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/dictionary.c -o crf/src/dictionary.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/holdout.c -o crf/src/holdout.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/logging.c -o crf/src/logging.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/params.c -o crf/src/params.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/quark.c -o crf/src/quark.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/rumavl.c -o crf/src/rumavl.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_arow.c -o crf/src/train_arow.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_averaged_perceptron.c -o crf/src/train_averaged_perceptron.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_l2sgd.c -o crf/src/train_l2sgd.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_lbfgs.c -o crf/src/train_lbfgs.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -O3 -c crf/src/train_passive_aggressive.c -o crf/src/train_passive_aggressive.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-8T8CYO/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c rcpp_crfsuite.cpp -o rcpp_crfsuite.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/home/mdowle/build/revdeplib/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-8T8CYO/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c RcppExports.cpp -o RcppExports.o
installing to /home/mdowle/build/revdeplib/crfsuite.Rcheck/00LOCK-crfsuite/00new/crfsuite/libs
** R
** data
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (crfsuite)

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/10/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 10.2.0-13ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-10-JvwpWM/gcc-10-10.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-JvwpWM/gcc-10-10.2.0/debian/tmp-gcn/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.0 (Ubuntu 10.2.0-13ubuntu1) 

crfsuite_0.3.4.tar.gz
./crfsuite.Rcheck/00check.log:* using log directory ‘/home/mdowle/build/revdeplib/crfsuite.Rcheck’
./crfsuite.Rcheck/00check.log:* using R version 4.0.3 (2020-10-10)
./crfsuite.Rcheck/00check.log:* using platform: x86_64-pc-linux-gnu (64-bit)
./crfsuite.Rcheck/00check.log:* using session charset: UTF-8
./crfsuite.Rcheck/00check.log:* checking for file ‘crfsuite/DESCRIPTION’ ... OK
./crfsuite.Rcheck/00check.log:* checking extension type ... Package
./crfsuite.Rcheck/00check.log:* this is package ‘crfsuite’ version ‘0.3.4’
./crfsuite.Rcheck/00check.log:* checking package namespace information ... OK
./crfsuite.Rcheck/00check.log:* checking package dependencies ... OK
./crfsuite.Rcheck/00check.log:* checking if this is a source package ... OK
./crfsuite.Rcheck/00check.log:* checking if there is a namespace ... OK
./crfsuite.Rcheck/00check.log:* checking for executable files ... OK
./crfsuite.Rcheck/00check.log:* checking for hidden files and directories ... OK
./crfsuite.Rcheck/00check.log:* checking for portable file names ... OK
./crfsuite.Rcheck/00check.log:* checking for sufficient/correct file permissions ... OK
./crfsuite.Rcheck/00check.log:* checking whether package ‘crfsuite’ can be installed ... WARNING
./crfsuite.Rcheck/00check.log:Found the following significant warnings:
./crfsuite.Rcheck/00check.log:  crf/src/crf1d_model.c:127:12: warning: writing 16 bytes into a region of size 4 [-Wstringop-overflow=]
./crfsuite.Rcheck/00check.log:See ‘/home/mdowle/build/revdeplib/crfsuite.Rcheck/00install.out’ for details.
./crfsuite.Rcheck/00check.log:* checking installed package size ... OK
./crfsuite.Rcheck/00check.log:* checking package directory ... OK
./crfsuite.Rcheck/00check.log:* checking ‘build’ directory ... OK
./crfsuite.Rcheck/00check.log:* checking DESCRIPTION meta-information ... OK
./crfsuite.Rcheck/00check.log:* checking top-level files ... OK
./crfsuite.Rcheck/00check.log:* checking for left-over files ... OK
./crfsuite.Rcheck/00check.log:* checking index information ... OK
./crfsuite.Rcheck/00check.log:* checking package subdirectories ... OK
./crfsuite.Rcheck/00check.log:* checking R files for non-ASCII characters ... OK
./crfsuite.Rcheck/00check.log:* checking R files for syntax errors ... OK
./crfsuite.Rcheck/00check.log:* checking whether the package can be loaded ... OK
./crfsuite.Rcheck/00check.log:* checking whether the package can be loaded with stated dependencies ... OK
./crfsuite.Rcheck/00check.log:* checking whether the package can be unloaded cleanly ... OK
./crfsuite.Rcheck/00check.log:* checking whether the namespace can be loaded with stated dependencies ... OK
./crfsuite.Rcheck/00check.log:* checking whether the namespace can be unloaded cleanly ... OK
./crfsuite.Rcheck/00check.log:* checking loading without being on the library search path ... OK
./crfsuite.Rcheck/00check.log:* checking dependencies in R code ... OK
./crfsuite.Rcheck/00check.log:* checking S3 generic/method consistency ... OK
./crfsuite.Rcheck/00check.log:* checking replacement functions ... OK
./crfsuite.Rcheck/00check.log:* checking foreign function calls ... OK
./crfsuite.Rcheck/00check.log:* checking R code for possible problems ... OK
./crfsuite.Rcheck/00check.log:* checking Rd files ... OK
./crfsuite.Rcheck/00check.log:* checking Rd metadata ... OK
./crfsuite.Rcheck/00check.log:* checking Rd cross-references ... OK
./crfsuite.Rcheck/00check.log:* checking for missing documentation entries ... OK
./crfsuite.Rcheck/00check.log:* checking for code/documentation mismatches ... OK
./crfsuite.Rcheck/00check.log:* checking Rd \usage sections ... OK
./crfsuite.Rcheck/00check.log:* checking Rd contents ... OK
./crfsuite.Rcheck/00check.log:* checking for unstated dependencies in examples ... OK
./crfsuite.Rcheck/00check.log:* checking contents of ‘data’ directory ... OK
./crfsuite.Rcheck/00check.log:* checking data for non-ASCII characters ... OK
./crfsuite.Rcheck/00check.log:* checking data for ASCII and uncompressed saves ... OK
./crfsuite.Rcheck/00check.log:* checking line endings in C/C++/Fortran sources/headers ... OK
./crfsuite.Rcheck/00check.log:* checking line endings in Makefiles ... OK
./crfsuite.Rcheck/00check.log:* checking compilation flags in Makevars ... OK
./crfsuite.Rcheck/00check.log:* checking for GNU extensions in Makefiles ... OK
./crfsuite.Rcheck/00check.log:* checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
./crfsuite.Rcheck/00check.log:* checking use of PKG_*FLAGS in Makefiles ... OK
./crfsuite.Rcheck/00check.log:* checking compiled code ... OK
./crfsuite.Rcheck/00check.log:* checking installed files from ‘inst/doc’ ... OK
./crfsuite.Rcheck/00check.log:* checking files in ‘vignettes’ ... OK
./crfsuite.Rcheck/00check.log:* checking examples ... OK
./crfsuite.Rcheck/00check.log:* checking for unstated dependencies in vignettes ... OK
./crfsuite.Rcheck/00check.log:* checking package vignettes in ‘inst/doc’ ... OK
./crfsuite.Rcheck/00check.log:* checking running R code from vignettes ... NONE
./crfsuite.Rcheck/00check.log:  ‘crfsuite-nlp.Rmd’ using ‘UTF-8’... OK
./crfsuite.Rcheck/00check.log:* checking re-building of vignette outputs ... OK
./crfsuite.Rcheck/00check.log:* checking PDF version of manual ... OK
./crfsuite.Rcheck/00check.log:* DONE
./crfsuite.Rcheck/00check.log:Status: 1 WARNING

as.crf

does not allow passing FUN, ...

make examples / vignette conditionally on package availability

Add crf attribute neighbouring features, integrate udpipe, add gui

need a better way of keeping the transformation function from raw text to enriched crf features

Difference in start & end position between shiny app and udpipe()

Great package, first of all !

I'm using the Shiny app to annotate custom entities in text. I have a very simple example where I load the text, and tag the a monetary amount in the text :

I then run the exact same piece of text through udpipe for automatic POS tagging. However, I'm unable to successfully join the udpipe output with the output from the Shiny app, because there is a difference in the start and end positions.

udpipe extarcts that "100,000,000.00" as a token, but with start and end position 1539-1552.

nchar(text) returns 1,839 characters, but the maximum end value in the dataframe returned by udpipe is 1911. So for some reason udpipe has 72 additional characters that are not present (or visible ?) in the text.

I have verified that the "text" and "text_visible" are the exact same
I have verified that the text is UTF-8 encoded before loading in the shiny app and before inputting into udpipe.
My versions of shiny and flexdashboard are 1.05 and 0.5.1.1, respectively.
I ran this on both Win 10 and Linux.

Any idea what might cause this ?

bug app newline: https://stackoverflow.com/questions/38364874/preserve-newlines-on-user-selected-text-currently-trying-with-window-getselecti

Seems like adding \r

typo

https://github.com/bnosac/crfsuite/blob/master/vignettes/crfsuite-nlp.Rmd#L98
should be
x <- x[, token_next := txt_sprintf("token[w+1]=%s", token_next), by = list(doc_id)] instead of
x <- x[, token_next := txt_sprintf("token[w-1]=%s", token_next), by = list(doc_id)]

vignette crfsuite-nlp not found

This doesn't execute:

vignette("crfsuite-nlp", package = "crfsuite")

Giving error:

vignette ‘crfsuite-nlp’ not found

allow to pass on an embedding matrix

Incorporate embeddings https://github.com/bnosac/crfsuite/blob/master/src/include/crfsuite_api.hpp#L147 at https://github.com/bnosac/crfsuite/blob/master/src/rcpp_crfsuite.cpp#L87

Wstrict-prototypes

Check Details
Version: 0.4
Check: whether package can be installed
Result: WARN
    Found the following significant warnings:
     ./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     ./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     crf/src/crf1d_encode.c:1031:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     crf/src/crf1d_feature.c:77:36: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     ./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     crf/src/params.c:252:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
     crf/src/quark.c:72:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Flavors: [r-devel-linux-x86_64-debian-clang](https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-clang/crfsuite-00check.html), [r-devel-linux-x86_64-fedora-clang](https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-clang/crfsuite-00check.html)

Version: 0.4
Check: whether package can be installed
Result: WARN
    Found the following significant warnings:
     crf/src/crfsuite_internal.h:169:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/params.h:36:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/crf1d_encode.c:1031:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/crf1d_feature.c:77:22: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/quark.h:39:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/params.c:252:20: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     crf/src/quark.c:72:10: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
Flavors: [r-devel-linux-x86_64-debian-gcc](https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-gcc/crfsuite-00check.html), [r-devel-linux-x86_64-fedora-gcc](https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-gcc/crfsuite-00check.html)


* installing *source* package ‘crfsuite’ ...
** package ‘crfsuite’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
make[1]: Entering directory '/data/gannet/ripley/R/packages/tests-clang/crfsuite/src'
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c cqdb/src/cqdb.c -o cqdb/src/cqdb.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c cqdb/src/lookup3.c -o cqdb/src/lookup3.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c liblbfgs/lib/lbfgs.c -o liblbfgs/lib/lbfgs.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_context.c -o crf/src/crf1d_context.o
In file included from crf/src/crf1d_context.c:46:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_encode.c -o crf/src/crf1d_encode.o
In file included from crf/src/crf1d_encode.c:45:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/crf1d_encode.c:47:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
crf/src/crf1d_encode.c:1031:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder()
                               ^
                                void
3 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_feature.c -o crf/src/crf1d_feature.o
In file included from crf/src/crf1d_feature.c:47:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
crf/src/crf1d_feature.c:77:36: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
static featureset_t* featureset_new()
                                   ^
                                    void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_model.c -o crf/src/crf1d_model.o
In file included from crf/src/crf1d_model.c:42:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_tag.c -o crf/src/crf1d_tag.o
In file included from crf/src/crf1d_tag.c:46:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crfsuite.c -o crf/src/crfsuite.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crfsuite_train.c -o crf/src/crfsuite_train.o
In file included from crf/src/crfsuite_train.c:43:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/crfsuite_train.c:44:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/dataset.c -o crf/src/dataset.o
In file included from crf/src/dataset.c:41:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/dictionary.c -o crf/src/dictionary.o
In file included from crf/src/dictionary.c:39:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/holdout.c -o crf/src/holdout.o
In file included from crf/src/holdout.c:41:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/logging.c -o crf/src/logging.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/params.c -o crf/src/params.o
In file included from crf/src/params.c:39:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
crf/src/params.c:252:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance()
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/quark.c -o crf/src/quark.o
In file included from crf/src/quark.c:37:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
crf/src/quark.c:72:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new()
                  ^
                   void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/rumavl.c -o crf/src/rumavl.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_arow.c -o crf/src/train_arow.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_averaged_perceptron.c -o crf/src/train_averaged_perceptron.o
In file included from crf/src/train_arow.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_arow.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
In file included from crf/src/train_averaged_perceptron.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_averaged_perceptron.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_l2sgd.c -o crf/src/train_l2sgd.o
In file included from crf/src/train_l2sgd.c:104:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_l2sgd.c:107:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_lbfgs.c -o crf/src/train_lbfgs.o
In file included from crf/src/train_lbfgs.c:46:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_lbfgs.c:49:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_passive_aggressive.c -o crf/src/train_passive_aggressive.o
In file included from crf/src/train_passive_aggressive.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_passive_aggressive.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang++ -std=gnu++11 -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic -frtti  -c rcpp_crfsuite.cpp -o rcpp_crfsuite.o
/usr/local/clang15/bin/clang++ -std=gnu++11 -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic -frtti  -c RcppExports.cpp -o RcppExports.o
/usr/local/clang15/bin/clang++ -std=gnu++11 -shared -L/usr/local/clang/lib64 -L/usr/local/clang15/lib -L/usr/local/lib64 -o crfsuite.so cqdb/src/cqdb.o cqdb/src/lookup3.o liblbfgs/lib/lbfgs.o crf/src/crf1d_context.o crf/src/crf1d_encode.o crf/src/crf1d_feature.o crf/src/crf1d_model.o crf/src/crf1d_tag.o crf/src/crfsuite.o crf/src/crfsuite_train.o crf/src/dataset.o crf/src/dictionary.o crf/src/holdout.o crf/src/logging.o crf/src/params.o crf/src/quark.o crf/src/rumavl.o crf/src/train_arow.o crf/src/train_averaged_perceptron.o crf/src/train_l2sgd.o crf/src/train_lbfgs.o crf/src/train_passive_aggressive.o rcpp_crfsuite.o RcppExports.o
rm -f cqdb/src/cqdb.o cqdb/src/lookup3.o liblbfgs/lib/lbfgs.o crf/src/crf1d_context.o crf/src/crf1d_encode.o crf/src/crf1d_feature.o crf/src/crf1d_model.o crf/src/crf1d_tag.o crf/src/crfsuite.o crf/src/crfsuite_train.o crf/src/dataset.o crf/src/dictionary.o crf/src/holdout.o crf/src/logging.o crf/src/params.o crf/src/quark.o crf/src/rumavl.o crf/src/train_arow.o crf/src/train_averaged_perceptron.o crf/src/train_l2sgd.o crf/src/train_lbfgs.o crf/src/train_passive_aggressive.o rcpp_crfsuite.o RcppExports.o 
make[1]: Leaving directory '/data/gannet/ripley/R/packages/tests-clang/crfsuite/src'
make[1]: Entering directory '/data/gannet/ripley/R/packages/tests-clang/crfsuite/src'
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c cqdb/src/cqdb.c -o cqdb/src/cqdb.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c cqdb/src/lookup3.c -o cqdb/src/lookup3.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c liblbfgs/lib/lbfgs.c -o liblbfgs/lib/lbfgs.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_context.c -o crf/src/crf1d_context.o
In file included from crf/src/crf1d_context.c:46:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_encode.c -o crf/src/crf1d_encode.o
In file included from crf/src/crf1d_encode.c:45:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/crf1d_encode.c:47:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
crf/src/crf1d_encode.c:1031:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder()
                               ^
                                void
3 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_feature.c -o crf/src/crf1d_feature.o
In file included from crf/src/crf1d_feature.c:47:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
crf/src/crf1d_feature.c:77:36: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
static featureset_t* featureset_new()
                                   ^
                                    void
1 warning generated.
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_model.c -o crf/src/crf1d_model.o
In file included from crf/src/crf1d_model.c:42:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crf1d_tag.c -o crf/src/crf1d_tag.o
In file included from crf/src/crf1d_tag.c:46:
In file included from ./crf/src/crf1d.h:37:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crfsuite.c -o crf/src/crfsuite.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/crfsuite_train.c -o crf/src/crfsuite_train.o
In file included from crf/src/crfsuite_train.c:43:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/crfsuite_train.c:44:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/dataset.c -o crf/src/dataset.o
In file included from crf/src/dataset.c:41:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/dictionary.c -o crf/src/dictionary.o
In file included from crf/src/dictionary.c:39:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/holdout.c -o crf/src/holdout.o
In file included from crf/src/holdout.c:41:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
1 warning generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/logging.c -o crf/src/logging.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/params.c -o crf/src/params.o
In file included from crf/src/params.c:39:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
crf/src/params.c:252:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance()
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/quark.c -o crf/src/quark.o
In file included from crf/src/quark.c:37:
./crf/src/quark.h:39:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new();
                  ^
                   void
crf/src/quark.c:72:19: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
quark_t* quark_new()
                  ^
                   void
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/rumavl.c -o crf/src/rumavl.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_arow.c -o crf/src/train_arow.o
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_averaged_perceptron.c -o crf/src/train_averaged_perceptron.o
In file included from crf/src/train_arow.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_arow.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_l2sgd.c -o crf/src/train_l2sgd.o
In file included from crf/src/train_averaged_perceptron.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_averaged_perceptron.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
In file included from crf/src/train_l2sgd.c:104:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_l2sgd.c:107:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_lbfgs.c -o crf/src/train_lbfgs.o
2 warnings generated.
/usr/local/clang15/bin/clang -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic  -c crf/src/train_passive_aggressive.c -o crf/src/train_passive_aggressive.o
In file included from crf/src/train_lbfgs.c:46:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_lbfgs.c:49:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang++ -std=gnu++11 -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic -frtti  -c rcpp_crfsuite.cpp -o rcpp_crfsuite.o
In file included from crf/src/train_passive_aggressive.c:44:
./crf/src/crfsuite_internal.h:169:32: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
encoder_t *crf1d_create_encoder();
                               ^
                                void
In file included from crf/src/train_passive_aggressive.c:46:
./crf/src/params.h:36:42: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
crfsuite_params_t* params_create_instance();
                                         ^
                                          void
2 warnings generated.
/usr/local/clang15/bin/clang++ -std=gnu++11 -I"/data/gannet/ripley/R/R-clang/include" -DNDEBUG -I./cqdb/include -I./crf/src -I./liblbfgs/include -I./include -I'/data/gannet/ripley/R/test-clang/Rcpp/include' -isystem /usr/local/clang15/include    -fpic  -O3 -Wall -pedantic -frtti  -c RcppExports.cpp -o RcppExports.o
make[1]: Leaving directory '/data/gannet/ripley/R/packages/tests-clang/crfsuite/src'
installing to /data/gannet/ripley/R/packages/tests-clang/crfsuite.Rcheck/00LOCK-crfsuite/00new/crfsuite/libs
** R
** data
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (crfsuite)

consider adding as_conll2002

as_conll2002 <- function(x, sep = "\t"){
  ## Each word has been put on a separate line and there is an empty line after each sentence.
  id <- udpipe::unique_identifier(x, fields = intersect(c("doc_id", "paragraph_id", "sentence_id"), colnames(x)))
  add <- ifelse(!duplicated(id), "\n\n", "\n")
  add[1] <- ""
  word   <- x$token
  entity <- x$chunk_entity
  sprintf("%s%s%s%s", add, word, sep, entity)
}
cat(as_conll2002(head(x, 100)))

to be used in training based on bertje

meaning of min_freq parameter

This question is not really related to your R code, but maybe you know the answer so let's try :)

was wondering what's the meaning of min_freq param, as in the documentation says it's a float. So I was always convinced it was a number in the range [0,1] (a percentage) but then I see you use 5.0.

Is that then the absolute frequency of a feature? (e.g. the number of times a feature appears in the training data)

Is it a requirement for the entire training set or per document?

Thanks!

Using other models than UDPipe

Hi again, I am back :)

I have finally created my own model using the example provided in the docs. This worked perfectly and no issues at all. However, when I use your RDRPOSTagger package to do tokenisation, I ran into weird behavior. The complete re-producable example is here.

Here are the issues/info:

1. x <- crf_cbind_attributes(x1, terms = c("upos", "lemma"), by = "doc_id") creates 68 cols in total when executed on UDPipe POS and tokenised dataframe whereas if you do the same for RDRPOSTagger POS and tokenised dataframe, it creates only 36 cols.
2. The chunk_entity col is not prefixed by I, B or O as it does for UDPipe tokenised and POS dataframe
3. It introduces duplicates for RDRPOSTagger tokenised dataframe but it doesn't duplicate if the UDPipe model is used.
4. You will have to change places when calling the merge(crfsuite_annotation_verbatim_to_annotate,rdr_tagging) method (in docs, you pass the annotated object first and then the y object) it throughs an error Error in merge.chunkrange(crfsuite_annotation_verbatim_to_annotate, rdr_tagging) : all(c(by.y, "start", "end") %in% colnames(y)) is not TRUE. But if you look at UDPipe tokenised dataframe. This also doesn't have start and end cols. I fixed it by changing the position of the method call by: x <- merge(rdr_tagging,crfsuite_annotation_verbatim_to_annotate)

I wrote a complete n00b getting starting guidehere.

thanks

valgrind issue

The valgrind 'additional issue' checks are now being done with Fedora 38
and its valgrind 3.21.0. This is flagging 'realloc with size 0' in packages

This is warned about in 'man realloc' on glibc and is said to be
non-portable and POSIX says

"If the size of the space requested is zero, the behavior shall be
implementation-defined ..."

giving alternatives (but macOS does something else again).

So please rewrite to avoid this (maybe use a wrapper that special-cases
size=0 ?)

Intent Classification

The suite of tools that you have built is fantastic, especially with the foresight to create a flexdashboard app to help tag entities in text. With that said, you mentioned the ability to do intent classification. Is the idea that the entire text is the chunk with the category applied? I am attempting to walk through your code examples in the README but my R session is hanging.

Obviously not a bug, but more of an information/example request.

add marginal for all labels

Tutorial

Looks like I hit a jackpot for creating my own models for NER, chunking etc.
A small tutorial would be ideal :)