bnosac / tokenizers.bpe Goto Github PK
View Code? Open in Web Editor NEWR package for Byte Pair Encoding based on YouTokenToMe
License: Mozilla Public License 2.0
R package for Byte Pair Encoding based on YouTokenToMe
License: Mozilla Public License 2.0
Fix R CMD check warning
using R Under development (unstable) (2023-01-01 r83536)
using platform: x86_64-pc-linux-gnu (64-bit)
R was compiled by
Debian clang version 15.0.6
GNU Fortran (Debian 12.2.0-10) 12.2.0
* running under: Debian GNU/Linux bookworm/sid
using session charset: UTF-8
checking for file ‘tokenizers.bpe/DESCRIPTION’ ... OK
checking extension type ... Package
this is package ‘tokenizers.bpe’ version ‘0.1.0’
package encoding: UTF-8
checking package namespace information ... OK
checking package dependencies ... OK
checking if this is a source package ... OK
checking if there is a namespace ... OK
checking for executable files ... OK
checking for hidden files and directories ... OK
checking for portable file names ... OK
checking for sufficient/correct file permissions ... OK
checking serialization versions ... OK
checking whether package ‘tokenizers.bpe’ can be installed ... WARNING
Found the following significant warnings:
youtokentome/cpp/bpe.cpp:1713:14: warning: unqualified call to 'std::move' [-Wunqualified-std-cast-call]
youtokentome/cpp/bpe.cpp:1725:14: warning: unqualified call to 'std::move' [-Wunqualified-std-cast-call]
See https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-clang/tokenizers.bpe-00install.html for details.
* used C++ compiler: ‘Debian clang version 15.0.6’
checking package directory ... OK
checking for future file timestamps ... OK
checking DESCRIPTION meta-information ... OK
checking top-level files ... OK
checking for left-over files ... OK
checking index information ... OK
checking package subdirectories ... OK
checking R files for non-ASCII characters ... OK
checking R files for syntax errors ... OK
checking whether the package can be loaded ... [1s/1s] OK
checking whether the package can be loaded with stated dependencies ... [0s/1s] OK
checking whether the package can be unloaded cleanly ... [0s/1s] OK
checking whether the namespace can be loaded with stated dependencies ... [0s/0s] OK
checking whether the namespace can be unloaded cleanly ... [1s/1s] OK
checking loading without being on the library search path ... [1s/1s] OK
checking use of S3 registration ... OK
checking dependencies in R code ... OK
checking S3 generic/method consistency ... OK
checking replacement functions ... OK
checking foreign function calls ... OK
checking R code for possible problems ... [5s/6s] OK
checking Rd files ... [0s/1s] OK
checking Rd metadata ... OK
checking Rd line widths ... OK
checking Rd cross-references ... OK
checking for missing documentation entries ... OK
checking for code/documentation mismatches ... OK
checking Rd \usage sections ... OK
checking Rd contents ... OK
checking for unstated dependencies in examples ... OK
checking contents of ‘data’ directory ... OK
checking data for non-ASCII characters ... [0s/1s] OK
checking LazyData ... OK
checking data for ASCII and uncompressed saves ... OK
checking line endings in C/C++/Fortran sources/headers ... OK
checking line endings in Makefiles ... OK
checking compilation flags in Makevars ... OK
checking for GNU extensions in Makefiles ... OK
checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
checking use of PKG_*FLAGS in Makefiles ... OK
checking use of SHLIB_OPENMP_*FLAGS in Makefiles ... OK
checking pragmas in C/C++ headers and code ... OK
checking compilation flags used ... OK
checking compiled code ... OK
checking examples ... [2s/3s] OK
checking PDF version of manual ... [5s/8s] OK
checking HTML version of manual ... [0s/1s] OK
checking for non-standard things in the check directory ... OK
DONE
Status: 1 WARNING
We can use e.g. data at https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/
Steps:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.