digikin / fluentd-elastic-kibana Goto Github PK
View Code? Open in Web Editor NEWWorking inital configuration for fluentd elastic and kibana
Working inital configuration for fluentd elastic and kibana
When I try to run this project, the fluentd
container fails to start. The error is the following:
/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
2023-01-28T14:08:16.859562828Z from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
...
Hi,
We're trying to set fluentd as the default docker driver using the example config using docker-compose (debian version for reasons explained at the bottom). I have installed elastic plugin (and systemd).
The problem is that fluentd logs itself and glitches with a bunch of \\, and I'm struggling to create a config that works.
I've tried using the FLUENT_LOG label, but logs keep showing in docker logs.
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<label @FLUENT_LOG>
<match fluent.*>
@type stdout
</match>
</label>
<match *.**>
@type copy
<store>
@type elasticsearch
host elasticsearch
user elastic
password *******
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
<buffer>
flush_interval 1s
</buffer>
</store>
<store>
@type stdout
</store>
</match>
A workaround would be to log this specific container to docker local driver, but I want fluentd to log itself.
As a side quest, I'm trying to mount /var/log/journal
to '/journal' and send to elastic the journal under a different index.
Any help would be appreciated.
Thanks!
Have a look at this issue: uken/fluent-plugin-elasticsearch#789
The gems for fluent-plugin-elasticsearch
and elasticsearch-xpack
are out of date in the base image.
Fought for a long while trying to get those features documented in 5.x to work, this was much of the reason.
I'd suggest updating the Dockerfile to something like:
FROM fluent/fluentd
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document"] # The current version is great
RUN ["gem", "install", "elasticsearch-xpack", "--version", "7.11.2"] # The image version is something like 5.x
# Copies a local elastic template file for use
# COPY ./conf/ecs-template.json /etc/fluentd/ecs-template.json
# Specifies a user with lesser privileges
#USER fluent
In docker compose file what is "3.5.2"
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "3.5.2"]
Is it fluent-plugin-elasticsearch version. If so, what is the latest version.?
I am having the exact setup suggested here. All containers except fluentd comes up. This is the logs from the fluentd container
2022-05-27 22:37:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-05-27 22:37:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
2022-05-27 22:37:04 +0000 [info]: gem 'fluentd' version '1.12.0'
/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/elasticsearch_fallback_selector.rb:1:in `<top (required)>'
from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `require_relative'
from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `<top (required)>'
from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:102:in `block in search'
Has anyone experienced this? How do I resolve this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.