digikin/fluentd-elastic-kibana

Working inital configuration for fluentd elastic and kibana

fluentd-elastic-kibana's Issues

fleuntd not starting

When I try to run this project, the fluentd container fails to start. The error is the following:

/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
2023-01-28T14:08:16.859562828Z 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
...

Filter fluentd logs issue

Hi,

We're trying to set fluentd as the default docker driver using the example config using docker-compose (debian version for reasons explained at the bottom). I have installed elastic plugin (and systemd).
The problem is that fluentd logs itself and glitches with a bunch of \\, and I'm struggling to create a config that works.
I've tried using the FLUENT_LOG label, but logs keep showing in docker logs.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<label @FLUENT_LOG>
  <match fluent.*>
    @type stdout
  </match>
</label>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    user elastic
    password *******
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    <buffer>
      flush_interval 1s
    </buffer>
  </store>
  <store>
    @type stdout
  </store>
</match>

A workaround would be to log this specific container to docker local driver, but I want fluentd to log itself.

As a side quest, I'm trying to mount /var/log/journal to '/journal' and send to elastic the journal under a different index.

Any help would be appreciated.
Thanks!

Version updates required in Dockerfile

Have a look at this issue: uken/fluent-plugin-elasticsearch#789

The gems for fluent-plugin-elasticsearch and elasticsearch-xpack are out of date in the base image.
Fought for a long while trying to get those features documented in 5.x to work, this was much of the reason.

I'd suggest updating the Dockerfile to something like:

FROM fluent/fluentd
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document"] # The current version is great
RUN ["gem", "install", "elasticsearch-xpack", "--version", "7.11.2"] # The image version is something like 5.x
# Copies a local elastic template file for use
# COPY ./conf/ecs-template.json /etc/fluentd/ecs-template.json
# Specifies a user with lesser privileges 
#USER fluent

In docker compose file what is "3.5.2"

In docker compose file what is "3.5.2"

RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "3.5.2"]
Is it fluent-plugin-elasticsearch version. If so, what is the latest version.?

fluentd container exiting immediately

I am having the exact setup suggested here. All containers except fluentd comes up. This is the logs from the fluentd container

2022-05-27 22:37:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-05-27 22:37:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
2022-05-27 22:37:04 +0000 [info]: gem 'fluentd' version '1.12.0'
/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
        from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/elasticsearch_fallback_selector.rb:1:in `<top (required)>'
        from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `require_relative'
        from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `<top (required)>'
        from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:102:in `block in search'

Has anyone experienced this? How do I resolve this?

Could not find a valid gem 'fluent-plugin-elasticsearch' (= 3.5.2)