rabbitmq/lz4-erlang

LZ4 compression library for Erlang.

LZ4 NIF Library for Erlang and Elixir

This is an LZ4 compression library for Erlang implemented as a NIF. It was originally developed for the needs of Looking Glass.

Project Maturity

This project is young and immature.

License and Copyright

See LICENSE.

(c) 2017-2019 Pivotal Software, Inc.

lz4-erlang's Issues

LZ4 1.8.1 has been released

https://github.com/lz4/lz4/releases/tag/v1.8.1

macos arm64 compatibility

hi, i try run it on my macos immediately when i saw #37 merged

i get the same error

this is my mix.exs deps

{:lz4, github: "rabbitmq/lz4-erlang", ref: "01765f8"}

compile log

/Users/watsy0007/code/chainindex/porter_oss/deps/lz4
compiling static library
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
compiling dynamic library 1.9.2
creating versioned links
cc -O3   -shared -fPIC -I../lib -DXXH_NAMESPACE=LZ4_  -c -o bench.o bench.c
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
cc -O3   -shared -fPIC -I../lib -DXXH_NAMESPACE=LZ4_  -c -o datagen.o datagen.c
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
cc -O3   -shared -fPIC -I../lib -DXXH_NAMESPACE=LZ4_  -c -o lz4cli.o lz4cli.c
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
cc -O3   -shared -fPIC -I../lib -DXXH_NAMESPACE=LZ4_  -c -o lz4io.o lz4io.c
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
cc -O3   -shared -fPIC -I../lib -DXXH_NAMESPACE=LZ4_  ../lib/lz4.o ../lib/lz4frame.o ../lib/lz4hc.o ../lib/xxhash.o bench.o datagen.o lz4cli.o lz4io.o -o lz4
make[1]: Nothing to be done for `all'.
 DEPEND host_triple.d
 ERLC   host_triple.erl
 APP    host_triple.app.src
 DEPEND lz4.d
 C      lz4.c
 C      lz4f.c
 C      nif_helpers.c
 LD     lz4_nif.so
ld: warning: ignoring file /Users/watsy0007/code/chainindex/porter_oss/deps/lz4/deps/lz4_src/lib/liblz4.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /Users/watsy0007/.asdf/installs/erlang/25.2/lib/erl_interface-5.3/lib/libei.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

error log

17:55:44.273 [debug] lz4: Trying to load "lz4_nif" from "/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/aarch64-apple-darwin22.2.0"
17:55:44.277 [debug] lz4: Couldn't load "lz4_nif" from "/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/aarch64-apple-darwin22.2.0":
{:error,
 {:load_failed,
  'Failed to load NIF library: \'dlopen(/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/aarch64-apple-darwin22.2.0/lz4_nif.so, 0x0002): tried: \'/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/a
arch64-apple-darwin22.2.0/lz4_nif.so\' (no such file), \'/System/Volumes/Preboot/Cryptexes/OS/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/aarch64-apple-darwin22.2.0/lz4_nif.so\' (no such file), \'/Users/watsy000
7/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/aarch64-apple-darwin22.2.0/lz4_nif.so\' (no such file)\''}}
17:55:44.278 [debug] lz4: Trying to load "lz4_nif" from "/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv"
17:55:44.278 [debug] lz4: Couldn't load "lz4_nif" from "/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv":
{:error,
 {:load_failed,
  'Failed to load NIF library: \'dlopen(/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so, 0x0002): tried: \'/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (mach-o file, b
ut is an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/System/Volumes/Preboot/Cryptexes/OS/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (no such file), \'/Users/watsy0007/code/chai
nindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (mach-o file, but is an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/Users/watsy0007/code/chainindex/porter_oss/deps/lz4/priv/lz4_nif.so\' (mach-o file, but is
 an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/System/Volumes/Preboot/Cryptexes/OS/Users/watsy0007/code/chainindex/porter_oss/deps/lz4/priv/lz4_nif.so\' (no such file), \'/Users/watsy0007/code/chainindex/porter_o
ss/deps/lz4/priv/lz4_nif.so\' (mach-o file, but is an incompatible architecture (have \'x86_64\', need \'arm64\'))\''}}
17:55:44.279 [warning] The on_load function for module lz4_nif returned:
{:error,
 {:load_failed,
  'Failed to load NIF library: \'dlopen(/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so, 0x0002): tried: \'/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (mach-o file, b
ut is an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/System/Volumes/Preboot/Cryptexes/OS/Users/watsy0007/code/chainindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (no such file), \'/Users/watsy0007/code/chai
nindex/porter_oss/_build/dev/lib/lz4/priv/lz4_nif.so\' (mach-o file, but is an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/Users/watsy0007/code/chainindex/porter_oss/deps/lz4/priv/lz4_nif.so\' (mach-o file, but is
 an incompatible architecture (have \'x86_64\', need \'arm64\')), \'/System/Volumes/Preboot/Cryptexes/OS/Users/watsy0007/code/chainindex/porter_oss/deps/lz4/priv/lz4_nif.so\' (no such file), \'/Users/watsy0007/code/chainindex/porter_o
ss/deps/lz4/priv/lz4_nif.so\' (mach-o file, but is an incompatible architecture (have \'x86_64\', need \'arm64\'))\''}}

sorry for my lack of c++ knowledge, Are there any steps I'm doing wrong?

failed to compile on ubuntu 18.04

OS: Ubuntu 18.04 LTS
Erl: Erlang/OTP 20 [erts-9.3.3.13]

my deps defined in rebar.config:
{deps, [
{syn, {git, "https://github.com/ostinelli/syn", {tag, "1.3.0"}}},
{lager, {git, "https://github.com/erlang-lager/lager", {branch, "master"}}},
{lz4, {git, "https://github.com/rabbitmq/lz4-erlang", {branch, "master"}}},
{looking_glass, {git, "https://github.com/rabbitmq/looking_glass.git", {branch, "master"}}},
rabbit
]}.

compile log:

===> Cleaning out goldrush...
===> Cleaning out lager...
===> Cleaning out looking_glass...
===> Cleaning out lz4...
===> Cleaning out syn...
===> Verifying dependencies...
===> Compiling syn
===> Compiling goldrush
===> Compiling lager
===> Compiling rabbit
_checkouts/rabbit/src/rabbit.erl:202: Warning: function default_restart_type/1 is unused

_checkouts/rabbit/src/rabbit_mnesia.erl:382: Warning: function is_registered_process_alive/1 is unused
_checkouts/rabbit/src/rabbit_mnesia.erl:682: Warning: function schema_info/1 is unused
_checkouts/rabbit/src/rabbit_mnesia.erl:686: Warning: function info/2 is unused

===> Compiling lz4
DEP host_triple
mv: cannot stat '/home/gy/relay/_build/default/lib/lz4/deps/host_triple/ebin/host_triple.app': No such file or directory
make[1]: Entering directory '/home/gy/relay/_build/default/lib/lz4/.erlang.mk/rebar'
./bootstrap
No beam files found.
Recompile: src/rebar
Recompile: src/rebar_abnfc_compiler
Recompile: src/rebar_app_utils
Recompile: src/rebar_appups
Recompile: src/rebar_asn1_compiler
Recompile: src/rebar_base_compiler
Recompile: src/rebar_cleaner
Recompile: src/rebar_config
Recompile: src/rebar_core
Recompile: src/rebar_cover_utils
Recompile: src/rebar_ct
src/rebar_ct.erl:291: Warning: crypto:rand_uniform/2 is deprecated and will be removed in a future release; use rand:uniform/1
Recompile: src/rebar_deps
Recompile: src/rebar_dia_compiler
Recompile: src/rebar_dialyzer
Recompile: src/rebar_edoc
Recompile: src/rebar_erlc_compiler
Recompile: src/rebar_erlydtl_compiler
Recompile: src/rebar_escripter
Recompile: src/rebar_eunit
src/rebar_eunit.erl:282: Warning: crypto:rand_uniform/2 is deprecated and will be removed in a future release; use rand:uniform/1
Recompile: src/rebar_file_utils
Recompile: src/rebar_getopt
Recompile: src/rebar_lfe_compiler
Recompile: src/rebar_log
Recompile: src/rebar_metacmds
Recompile: src/rebar_mustache
Recompile: src/rebar_neotoma_compiler
Recompile: src/rebar_otp_app
Recompile: src/rebar_otp_appup
Recompile: src/rebar_port_compiler
Recompile: src/rebar_proto_compiler
Recompile: src/rebar_proto_gpb_compiler
Recompile: src/rebar_protobuffs_compiler
Recompile: src/rebar_qc
Recompile: src/rebar_rand_compat
Recompile: src/rebar_rel_utils
Recompile: src/rebar_reltool
Recompile: src/rebar_require_vsn
Recompile: src/rebar_shell
Recompile: src/rebar_subdirs
Recompile: src/rebar_templater
Recompile: src/rebar_upgrade
Recompile: src/rebar_utils
Recompile: src/rebar_xref
Recompile: src/rmemo
==> rebar (compile)
==> rebar (escriptize)
Congratulations! You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps.
make[1]: Leaving directory '/home/gy/relay/_build/default/lib/lz4/.erlang.mk/rebar'
/home/gy/relay/_build/default/lib/lz4
Error: No Makefile to build dependency /home/gy/relay/_build/default/lib/lz4/deps/lz4_src.
erlang.mk:4183: recipe for target 'deps' failed
make: *** [deps] Error 2
===> Hook for compile failed!

erlang.mk upgrade for Erlang 23 compatibility

Looks like -lerl_interface is not available in OTP 23. We should bump erlang.mk which removes its use:

• ninenines/erlang.mk@2331c34
• ninenines/erlang.mk@ada293a

@dumbbell @essen do we have to keep erlang.mk in sync for standalone libraries such as this one (or Looking Glass, or Ra which depends on Looking Glass)?

Rebar3 compatibility

Hello,

I have a project with rebar3 and this lib as dep.

when i try to do rebar3 upgrade lz4 I receive:

===> Dependency failure: source for lz4 does not contain a recognizable project and can not be built

This happens starting with revision "851d9801e97b43f0c3473b91547988f21bc9d380" older one works just fine

Silviu

Support multiple architectures with multiple copies of the native code

In order to support multiple architectures in a single package, I would like be able to ship multiple copies of the NIF shared library and the Erlang code should load the appropriate one.

We could search for the native library in two places:

1. In priv/$arch/lz4_nif.so
2. In priv/lz4_nif.so

What I have in mind is based on the usual architecture triples used by Autoconf (exposed by Erlang) or LLVM to give a few examples.

The one exposed by Erlang is the one set by the configure script when Erlang was compiled:

1> erlang:system_info(system_architecture).
"amd64-portbld-freebsd13.0"

Unfortunately, there is no consistency in that triple: it is very dependent on the toolchain and user configuration at the time Erlang was compiled. So several values could mean the same architecture.

For instance on FreeBSD:

• The official Erlang package exposes amd64-portbld-freebsd13.0
• Erlang built from source in my home directory exposes x86_64-unknown-freebsd13.0

I prefer what LLVM does: the triple is normalized. Here is the list exposed by Rust, which uses LLVM internally:

$ rustc --print target-list
aarch64-fuchsia
aarch64-linux-android
aarch64-pc-windows-msvc
aarch64-unknown-cloudabi
aarch64-unknown-freebsd
...
arm-linux-androideabi
arm-unknown-linux-gnueabi
arm-unknown-linux-gnueabihf
arm-unknown-linux-musleabi
arm-unknown-linux-musleabihf
armebv7r-none-eabi
armebv7r-none-eabihf
armv4t-unknown-linux-gnueabi
armv5te-unknown-linux-gnueabi
armv5te-unknown-linux-musleabi
armv6-unknown-freebsd
armv6-unknown-netbsd-eabihf
armv7-linux-androideabi
armv7-unknown-cloudabi-eabihf
...
armv7r-none-eabi
armv7r-none-eabihf
asmjs-unknown-emscripten
hexagon-unknown-linux-musl
i586-pc-windows-msvc
i586-unknown-linux-gnu
i586-unknown-linux-musl
i586-wrs-vxworks
i686-apple-darwin
i686-linux-android
i686-pc-windows-gnu
i686-pc-windows-msvc
i686-unknown-cloudabi
...
x86_64-unknown-openbsd
x86_64-unknown-redox
x86_64-unknown-uefi
x86_64-uwp-windows-gnu
x86_64-wrs-vxworks

So what do you think if:

• We modify the native code loading to look into both directories;
• We normalize the triple returned by Erlang to an LLVM one to format the directory name?

On my laptop, this would give:

1. amd64-portbld-freebsd13.0 -> x86_64-unknown-freebsd
2. Try to load priv/x86_64-unknown-freebsd/lz4_nif.so
3. Try to load priv/lz4_nif.so

Taking care of building all the copies and shipping them is not in scope for this issue.