ixixi / fluent-plugin-sns Goto Github PK
View Code? Open in Web Editor NEWAmazon SNS output plugin for Fluent event collector
Amazon SNS output plugin for Fluent event collector
Hi
I would like to provide access to fluentd access to sns by using a Kubernetes service account .
see was doc:
https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
that will provide the pod a AWS_WEB_IDENTITY_TOKEN_FILE.
aws cli - works using this method but it looks fluentd and sns plugin can't use this method and need
aws key and secret.
sudo td-agent-gem install fluent-plugin-sns
td-agent 0.12.20
sudo /etc/init.d/td-agent status
td-agent is running
curl -X POST -d 'json={"json":"message"}' http://127.0.0.1:8827/test <======= Failed
500 Internal Server Error
undefined method `publish' for nil:NilClass
curl -X POST -d 'json={"json":"message"}' http://127.0.0.1:8827/debug.test <==== Successful
## match tag=debug.** and dump to console
<match debug.**>
type stdout
</match>
<match test>
type sns
sns_topic_name XXXX
aws_key_id XXX
aws_sec_key XXX
</match>
# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
<source>
type http
port 8827
</source>
2016-05-02 09:37:24 +0000 [info]: listening fluent socket on 0.0.0.0:24224 2016-05-02 09:37:24 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine" 2016-05-02 09:37:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `publish' for nil:NilClass" tag="test" 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sns-2.1.7/lib/fluent/plugin/out_sns.rb:71:in `block in emit' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:54:in `call' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:54:in `each' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sns-2.1.7/lib/fluent/plugin/out_sns.rb:67:in `emit' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event_router.rb:88:in `emit_stream' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event_router.rb:79:in `emit' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:186:in `on_request' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:353:in `call' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:353:in `on_message_complete' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:255:in `<<' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:255:in `on_read' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:123:in `on_readable' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:186:in `on_readable' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run_once' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run' 2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:122:in `run' <==== Failed 2016-05-02 09:37:50 +0000 debug.test: {"json":"message"} <==== Successful
I was able to list the topics on SNS on ec2 as well so no issues with credentials
AWS Access Key ID [****************XXXX]: AWS Secret Access Key [****************XXXX]: Default region name [us-east-1]: Default output format [json]:
{ "Topics": [ { "TopicArn": "arn:aws:sns:us-east-1:XXX" }, { "TopicArn": "arn:aws:sns:us-east-1:XXX" } ] }
Currently fluentd v0.10 is required, would be very useful if newer fluentd versions were supported.
SNS Message Attributes provides support for sending additional metadata about the message.
It seems that this is only available in the AWS SDK v2, so a migration to the newer SDK is needed (v1 is used ATM).
I'm trying to upload individual log entries as messages to SNS, but I'm getting this error message:
emit transaction failed: error_class=NoMethodError error="undefined method `publish' for nil:NilClass" tag="public-api.access" and a stack trace, whenever an entry to the log is added.
Here's my config:
Configuration:
<match http_source>
type sns
sns_topic_name XX
aws_key_id XX
aws_sec_key XX
sns_endpoint XX
<secondary>
type file
path /var/log/td-agent/failed_events_http_source
</secondary>
</match>
# Accept events from HTTP source at http://SOMEURL:8827/http_source/<JSON EVENT>
<source>
type http
port 8827
</source>
Steps to reproduce:
Expectation:
Events should now be appended to /var/log/td-agent/failed_events_http_source
Outcome:
On console:
500 Internal Server Error
Topic does not exist
Reference: http://docs.fluentd.org/articles/buffer-plugin-overview#secondary-output
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.