indix / kafkajs-lz4 Goto Github PK

View Code? Open in Web Editor NEW

14.0 20.0 8.0 376 KB

🗜 LZ4 compression codec for KafkaJS

License: MIT License

JavaScript 16.98% TypeScript 70.60% Shell 12.42%

node typescript kafka lz4 indix

kafkajs-lz4's Introduction

kafkajs-lz4

TypeScript-ready lz4 compression codec for KafkaJS.

ℹ️ Requires Node v10 or above to work.

Install

$ yarn add kafkajs-lz4

Usage

import { CompressionTypes, CompressionCodecs } from 'kafkajs';
import LZ4Codec from 'kafkajs-lz4';

CompressionCodecs[CompressionTypes.LZ4] = new LZ4Codec().codec;

Options

All options are transparently passed on to the lz4-asm library's compress options.

Example

To set the highest level of compression for your Kafka messages —

const lz4Codec = new LZ4Codec({
    preferences: {
        compressionLevel: 16
    }
}).codec;

CompressionCodecs[CompressionTypes.LZ4] = lz4Codec;

kafkajs-lz4's People

Contributors

Stargazers

Watchers

Forkers

saanuregh utkarshpaliwal9 palmy523 whizzzkid freakachoo captainup chance722 ivagafonov

kafkajs-lz4's Issues

install broken since lz4 updated to 6.0.4

OSX 10.13.6
node v12.18.2
yarn v1.22.4

Can't install kafkajs-lz4 since it now pulls lz4 6.0.4 which is broken.

I get this error after yarn add kafkajs-lz4

gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
CXX(target) Release/obj.target/lz4/lib/binding/lz4_binding.o
../lib/binding/lz4_binding.cc:8:10: fatal error: '../../deps/lz4/lib/lz4.h' file not found
#include "../../deps/lz4/lib/lz4.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/lz4/lib/binding/lz4_binding.o] Error 1
gyp ERR! build error

Upgrade dependencies for vulnerabilities

Vulnerability: Regular Expression Denial of Service (ReDoS)

Introduced through: [email protected] > [email protected] > [email protected] > [email protected] > [email protected] > [email protected]

Vulnerable module : ansi-regex
Introduced through [email protected]
Fixed in [email protected], 4.1.1, 5.0.1, 6.0.1

Ref: https://security.snyk.io/vuln/SNYK-JS-ANSIREGEX-1583908

Unable to import

Seems you have mixed the default import vs module.export.
export default class LZ4Codec
vs
module.exports = LZ4Codec;
So unable to use the library.
Can we please fixed it?

RuntimeError: abort(OOM). Build with -s ASSERTIONS=1 for more info.

OOM
RuntimeError: abort(OOM). Build with -s ASSERTIONS=1 for more info.
at y (D:\LTXY\services\hainan\kafka_to_databus\node_modules\lz4-asm\dist\lz4wasm.js:16:393)
at c (D:\LTXY\services\hainan\kafka_to_databus\node_modules\lz4-asm\dist\lz4wasm.js:19:185)
at :wasm-function[8]:0x2393
at :wasm-function[6]:0x139d
at :wasm-function[37]:0x11419
at m (:wasm-function[58]:0x15c37)
at b.L (D:\LTXY\services\hainan\kafka_to_databus\node_modules\lz4-asm\dist\lz4wasm.js:36:268)
at b.K (D:\LTXY\services\hainan\kafka_to_databus\node_modules\lz4-asm\dist\lz4wasm.js:36:340)
at Object.decompress (D:\LTXY\services\hainan\kafka_to_databus\node_modules\lz4-asm\dist\lz4wasm.js:41:180)
at D:\LTXY\services\hainan\kafka_to_databus\node_modules\kafkajs-lz4\lib\index.js:37:50

Does not work on node 18

Explanation here ukyo/lz4.js#14 (comment)

Error for Google index: TypeError: Failed to parse URL from /Users/mark/Phpstorm/dtq/node_modules/lz4-asm/dist/_lz4.wasm

npm package appears corrupt

Unable to install this package:

 yarn add --registry https://registry.npmjs.org/ kafkajs-lz4
yarn add v1.22.4
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error https://registry.npmjs.org/lz4/-/lz4-0.6.3.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, link '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/build/Release/lz4.node' -> '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/lz4.node'"
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Snyk Error because of [email protected] : CVSS 7.5 - High Severity

Snyk Error because of [email protected]

Snyk Issue: https://security.snyk.io/vuln/SNYK-JS-ANSIREGEX-1583908?_gl=1%2awvv85p%2a_ga%2aMTIyOTcxMDgyLjE3MDgzNDg0NTI.%2a_ga_X9SH3KP7B4%2aMTcxMTI5ODM3NC4yNi4xLjE3MTEyOTg5MzAuMC4wLjA.

yarn add kafkajs-lz4 issue

Hello everyone,

I end up with this issue when trying install kafkajs-lz4:

Debugger listening on ws://127.0.0.1:35199/f902c608-883d-4c84-963a-d05c60fb4e8c
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
yarn add v1.22.5
warning ../../package.json: No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /home/mickael/certificate/kafkaConnection/node_modules/lz4: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: /home/mickael/certificate/kafkaConnection/node_modules/lz4
Output:
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info find Python using Python version 2.7.18 found at "/usr/bin/python2"
gyp info spawn /usr/bin/python2
gyp info spawn args [
gyp info spawn args   '/home/mickael/.nvm/versions/node/v12.18.4/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/mickael/certificate/kafkaConnection/node_modules/lz4/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/mickael/.nvm/versions/node/v12.18.4/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/mickael/.cache/node-gyp/12.18.4/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/mickael/.cache/node-gyp/12.18.4',
gyp info spawn args   '-Dnode_gyp_dir=/home/mickael/.nvm/versions/node/v12.18.4/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/mickael/.cache/node-gyp/12.18.4/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/mickael/certificate/kafkaConnection/node_modules/lz4',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
Debugger listening on ws://127.0.0.1:42773/b91d4216-5886-4dc8-a6be-86c4cb20d334
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Waiting for the debugger to disconnect...
gyp: Call to 'node -e "require('nan')"' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/home/mickael/.nvm/versions/node/v12.18.4/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
gyp ERR! System Linux 5.4.0-48-generic
gyp ERR! command "/home/mickael/.nvm/versions/node/v12.18.4/bin/node" "/home/mickael/.nvm/versions/node/v12.18.4/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/mickael/certificate/kafkaConnection/node_modules/lz4
gyp ERR! node -v v12.18.4
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok

Did anyone face this issue?
Is there a way to work around it?
Thanks in advance for your answers