Not work with zmq and plainzmq

[driskell]

Not work with zmq and plainzmq

[driskell]

Hi @scakkia

Best to omit secret keys and such in public posts.

What do the Logstash logs look like? It appears there may be a problem at that side.

Jason

[driskell]

Also - to load balance you will need to set multiple ZMQ servers. I presume you have specific just the one here for testing?

[scakkia]

Hi Jason,
thanks for looking

sorry for key, it's only devel... and yes I'm use only one server for testing.
I've already running 2 instance of log-courier, with 2 logstash indexer in "tls" under haproxy, but I don't like haproxy because it not re-balance after put one LS down and up.

Probably I missing something on LS, I use your howto https://github.com/driskell/log-courier/blob/develop/docs/LogstashIntegration.md

Perhaps something relative to #53 ?

LS LOG:

/opt/logstash/bin/logstash -v -v  -l /tmp/test.log -f /etc/logstash/api-zmq.conf 
The -vv flag is deprecated and will be removed in a future release. You should use --debug instead. {:level=>:warn}
Sending logstash logs to /tmp/test.log.
Reading config file {:file=>"logstash/agent.rb", :level=>:debug, :line=>"301"}
Compiled pipeline code:
@inputs = []
@filters = []
@outputs = []
@input_courier_1 = plugin("input", "courier", LogStash::Util.hash_merge_many({ "transport" => ("zmq".force_encoding("UTF-8")) }, { "port" => 6001 }, { "curve_secret_key" => ("XXXXXXXXXXXXX)".force_encoding("UTF-8")) }))

@inputs << @input_courier_1
@filter_mutate_2 = plugin("filter", "mutate", LogStash::Util.hash_merge_many({ "add_tag" => [("api-nginx-access".force_encoding("UTF-8"))] }))

@filters << @filter_mutate_2
@output_stdout_3 = plugin("output", "stdout", LogStash::Util.hash_merge_many({ "codec" => ("rubydebug".force_encoding("UTF-8")) }))

@outputs << @output_stdout_3
  @filter_func = lambda do |event, &block|
    extra_events = []
    @logger.debug? && @logger.debug("filter received", :event => event.to_hash)
    if ((event["[type]"] == ("api-nginx-access".force_encoding("UTF-8"))))
      newevents = []
      extra_events.each do |event|
        @filter_mutate_2.filter(event) do |newevent|
          newevents << newevent
        end
      end
      extra_events += newevents
      @filter_mutate_2.filter(event) do |newevent|
        extra_events << newevent
      end
      if event.cancelled?
        extra_events.each(&block)
        return
      end

    end

    extra_events.each(&block)
  end
  @output_func = lambda do |event, &block|
    @logger.debug? && @logger.debug("output received", :event => event.to_hash)
    @output_stdout_3.handle(event)

  end {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"26"}
Using milestone 1 input plugin 'courier'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones {:level=>:warn, :file=>"logstash/config/mixin.rb", :line=>"209"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@transport = "zmq" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@port = 6001 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@curve_secret_key = "XXXXXXXX)" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@debug = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@host = "0.0.0.0" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@ssl_verify = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::Courier/@ssl_verify_default_ca = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@add_tag = ["api-nginx-access"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Filters::Mutate/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug > {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}

thx

[driskell]

Hi @scakkia - can you regenerate keys with lc-curvekey and try again? I think I saw this before when testing when once of the key sets did not match.

Specifically, in the log-courier side, public key and secret key must be a valid pair. Then the server key must be the public equivalent of the Logstash secret key.

There's an oddity in ZMQ where it doesn't report back Curve problems, which I've yet to investigate and improve.

[scakkia]

Hi @driskell
I regenerate keys with lc-curvekey, but nothing has changed.
The problem also occurs with "plainzmq", so without any key/encryption.

I build and install libsodium and zeromq from tarball:

http://download.libsodium.org/libsodium/releases/LATEST.tar.gz (1.0.0)
./configure
make && make check
make install
ldconfig

http://download.zeromq.org/zeromq-4.0.4.tar.gz
./configure --with-libsodium
make && make check
make install
ldconfig

and then build log-courier with zmq4 (also try zmq3)

make with=zmq4
GOPATH: /opt/log-courier
go get -d -tags " zmq zmq_4_x" log-courier
go install -tags " zmq zmq_4_x" log-courier
go get -d -tags " zmq zmq_4_x" lc-tlscert
go install -tags " zmq zmq_4_x" lc-tlscert
go get -d -tags " zmq zmq_4_x" lc-admin
go install -tags " zmq zmq_4_x" lc-admin
go get -d -tags " zmq zmq_4_x" lc-curvekey
go install -tags " zmq zmq_4_x" lc-curvekey

with GO version:
go version go1.3 linux/amd64

thanks for your interest

[scakkia]

Hi @driskell
I'm just try on fresh install of Ubuntu 14.04:

1. with packages mentioned above, log-courier build "with=zmq4", key generate with lc-curvekey and transport "zmq"
2. with libzmq3 (4.0.4+dfsg-2) system packages, log-courier build "with=zmq3" and transport "plainzmq"

but unfortunately it never worked.

[driskell]

Hi scakkia,

libzmq3 (4.0.4) is actually ZMQ4. But that's fine. Building using with=zmq4 is also the same as with=zmq3 - the only difference is whether "zmq" is available. with=zmq3 only makes "plainzmq" available. But to get things working, I would build with=zmq4 and try get it working using "plainzmq", then add the curve on top once working.

Saying that, I've tried to reproduce this and wasn't able to. I did find one issue with zmq transport - where if you had network timeout error it could hang and not send anymore logs. I've fixed that now in develop, so you could try that version and see if anything different.

Jason

[scakkia]

Hi Jason,

thanks for watching.

I build with=zmq4 and try get it working using "plainzmq", but not work.
I installed log-courier on same vm of logstash, so no timeout problem.

In logstash debug log this line is repeated a lot of time:
{:timestamp=>"2014-10-16T23:08:56.573000+0200", :message=>"[LogCourierServer] Invalid message: multipart unexpected", :level=>:warn, :file=>"log-courier/server_zmq.rb", :line=>"127"}

and almost at the same time in log-courier log I see:
"Transport error, will try again: Server did not respond within network timeout"

I can reproduce this issue every time on fresh Debian 7.6, zeromq 4.0.5, golang 1.3.

Here's what I've done from a clean install of debian .... probably I'm missing something .. but I'm not understanding what :-(

Install logstash

echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | tee /etc/apt/sources.list.d/logstash.list
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
apt-get update -y
apt-get install logstash unzip curl openjdk-7-jre-headless

Install some Packages

apt-get install git make g++ pkg-config

Install GO 1.3

apt-get install devscripts build-essential
apt-get build-dep golang-go

wget http://ftp.de.debian.org/debian/pool/main/g/golang/golang_1.3-3.dsc
wget http://ftp.de.debian.org/debian/pool/main/g/golang/golang_1.3.orig.tar.gz
wget http://ftp.de.debian.org/debian/pool/main/g/golang/golang_1.3-3.debian.tar.xz

dpkg-source -x golang_1.3-3.dsc
cd golang-1.3/
debuild -us -uc
cd ..
dpkg -i
golang-go_1.3-3_amd64.deb
golang-src_1.3-3_amd64.deb
golang-go-linux-amd64_1.3-3_amd64.deb
vim-syntax-go_1.3-3_all.deb

Install ZMQ and log-courier

wget http://download.zeromq.org/zeromq-4.0.5.tar.gz
tar -xzvf zeromq-4.0.5.tar.gz
cd zeromq-4.0.5
./configure
make && make check
make install
ldconfig

git clone https://github.com/driskell/log-courier
cd log-courier
make with=zmq4

Configure Logstash:

cd /opt/log-courier
make gem

cd /opt/logstash
export GEM_HOME=vendor/bundle/jruby/1.9
java -jar vendor/jar/jruby-complete-1.7.11.jar -S gem install /opt/log-courier/log-courier-0.15.gem

Fetching: ffi-rzmq-core-1.0.3.gem (100%)
Successfully installed ffi-rzmq-core-1.0.3
Fetching: ffi-rzmq-2.0.1.gem (100%)
Successfully installed ffi-rzmq-2.0.1
Successfully installed log-courier-0.15
3 gems installed

cd /opt/log-courier
cp -rvf lib/logstash /opt/logstash/lib

/etc/logstash/zmq.conf
input {
courier {
transport => "plainzmq"
port => 4000
}
}
filter {
if [type] == "api-nginx-access" {
mutate {
add_tag => [ "api-nginx-access" ]
}
}
}
output {
stdout { codec => rubydebug }
}

Start Logstash:

/opt/logstash/bin/logstash --debug -l /tmp/test.log -f /etc/logstash/zmq.conf

Configure and start log-courier:

log-courier.conf
{
"general": {
"admin enabled": true,
"admin listen address": "tcp:127.0.0.1:1234",
"log level": "debug",
"log file": "/var/log/log-courier.log"
},

"network": {
"transport": "plainzmq",
"servers": [ "localhost:4000" ],
"timeout": 5
},

"files": [
{
"paths": [ "/opt/nginx-access.log" ],
"fields": { "type": "api-nginx-access" }
}
]
}

/opt/log-courier/bin/log-courier -config=/opt/log-courier.conf

thx

[driskell]

Hi scakkia,

I found an issue in 0.15 that meant after a network timeout on ZMQ it would hang log-courier. Could you try the latest develop branch and see if the issue is fixed?

Thanks

Jason

[driskell]

If you continue to have issues in the latest dev version (will be released as 0.16 soon) - reopen and let me know.

[scakkia]

Hi @driskell
I can confirm that the issue is resolved with the last sable version 1.1.