wattgo/node-kafka0.8

Kafka 0.8 client for node

node-kafka0.8's Issues

Error on npm install

I add this to my packages.json file :
"kafka0.8":"*"

but when I try to run npm install (I'm on windows 7), I get several errors :
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
bignum.cc
snappy-sinksource.cc
snappy-stubs-internal.cc
snappy.cc
hiredis.c
async.c
net.c
....\deps\hiredis\hiredis.c(35): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\no
de_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
sds.c
....\deps\hiredis\net.c(35): fatal error C1083: Cannot open include file: 'sys/socket.h': No such file or directory [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\no
de_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
....\deps\hiredis\async.c(35): fatal error C1083: Cannot open include file: 'strings.h': No such file or directory [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\nod
e_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(45): error C2054: expected '(' to follow 'inline' [C:\User
s\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(45): error C2085: 'sdslen' : not in formal parameter list
[C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(45): error C2143: syntax error : missing ';' before '{' [C
:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(50): error C2054: expected '(' to follow 'inline' [C:\User
s\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(50): error C2085: 'sdsavail' : not in formal parameter lis
t [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
c:\users\yoann.diguet\dropbox\node.js\kafkafailover\node_modules\kafka0.8\node_modules\hiredis\deps\hiredis\sds.h(50): error C2143: syntax error : missing ';' before '{' [C
:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis\build\deps\hiredis.vcxproj]
....\deps\hiredis\sds.c(179): warning C4013: 'va_copy' undefined; assuming extern returning int [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8
node_modules\hiredis\build\deps\hiredis.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: C:\Program Files (x86)\MSBuild\12.0\bin\msbuild.exe failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:267:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\hiredis
gyp ERR! node -v v0.10.13
gyp ERR! node-gyp -v v0.10.2
gyp ERR! not ok
unbuild [email protected]
unbuild [email protected]
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\xlocale(323): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
[C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\buffermaker\node_modules\bignum\build\bignum.vcxproj]
snappy.vcxproj -> C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\Release\snappy.lib
binding.cc
..\bignum.cc(10): fatal error C1083: Cannot open include file: 'openssl/bn.h': No such file or directory [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\k
afka0.8\node_modules\buffermaker\node_modules\bignum\build\bignum.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: C:\Program Files (x86)\MSBuild\12.0\bin\msbuild.exe failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:267:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files (x86)\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "configure" "build"
gyp ERR! cwd C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\buffermaker\node_modules\bignum
gyp ERR! node -v v0.10.13
gyp ERR! node-gyp -v v0.10.2
gyp ERR! not ok
unbuild [email protected]
npm ERR! error rolling back Error: ENOTEMPTY, rmdir 'C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\buffermaker\node_modules\bignum\b
uild\Release\obj'
npm ERR! error rolling back [email protected] { [Error: ENOTEMPTY, rmdir 'C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\buffermaker\no
de_modules\bignum\build\Release\obj']
npm ERR! error rolling back errno: 53,
npm ERR! error rolling back code: 'ENOTEMPTY',
npm ERR! error rolling back path: 'C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\buffermaker\node_modules\bignum\build
\Release\obj' }
npm ERR! weird error 1
unbuild [email protected]
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\xlocale(323): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
[C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\binding.vcxproj]
C:\Users\yoann.diguet.node-gyp\0.10.13\deps\v8\include\v8.h(179): warning C4506: no definition for inline function 'v8::Persistent v8::Persistent::New(v8::Handle)
' [C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\binding.vcxproj]
with
[
T=v8::Object
]
Creating library C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\Release\binding.lib and object C:\Users\yoann.digu
et\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\Release\binding.exp
Generating code
Finished generating code
binding.vcxproj -> C:\Users\yoann.diguet\Dropbox\node.js\KafkaFailover\node_modules\kafka0.8\node_modules\snappy\build\Release\binding.node
npm ERR! not ok code 0

Java Snappy Framing

snappy-java used by Kafka is implementing framing format
We should handle this
See : http://snappy.googlecode.com/svn/trunk/framing_format.txt

UTF-8 Encoding?

When using UTF-8 encoding to support other languages such as Korean, messages that are placed in the Kafka queue seem to have the incorrect offset. When consuming the UTF-8 message out of the queue for every UTF-8 encoded character 2 bytes are missing off the end. The message looks good in the Kafka logs but when they are consumed that are missing x bytes off the end.

Support consumer group (groupId) on consumer.

Implement consumer group (groupId) support such that a consumer group can load balance through zookeeper without each consumer needing to explicitly subscribe to a partition.

Basically the equivalent of this console command.

kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic --group console-group

balance messages between multiple consumers on a specific topic

hi,

can i balance/distribute messages to multiples consumers having the same group?
i would like to send message in a topic to multiple consumers but i would that every message will be delivered to one consumer and not to all?

cordially