ixixi/fluent-plugin-sns

Amazon SNS output plugin for Fluent event collector

fluent-plugin-sns's Issues

sns pluging not working when using fluentd with sa include aws role arn

Hi
I would like to provide access to fluentd access to sns by using a Kubernetes service account .
see was doc:
https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
that will provide the pod a AWS_WEB_IDENTITY_TOKEN_FILE.

aws cli - works using this method but it looks fluentd and sns plugin can't use this method and need
aws key and secret.

Unable to post to SNS when attempted from Amazon EC2

Installed plugin using

sudo td-agent-gem install fluent-plugin-sns

td-agent --version

td-agent 0.12.20

Console output

sudo /etc/init.d/td-agent status
td-agent is running

curl -X POST -d 'json={"json":"message"}' http://127.0.0.1:8827/test <======= Failed
500 Internal Server Error
undefined method `publish' for nil:NilClass

curl -X POST -d 'json={"json":"message"}' http://127.0.0.1:8827/debug.test <==== Successful

td-agent.conf

## match tag=debug.** and dump to console                                                                                                                                            
<match debug.**>
  type stdout
</match>

<match test>
  type sns
  sns_topic_name XXXX
  aws_key_id XXX
  aws_sec_key XXX
</match>

# HTTP input                                                                                                                                                                         
# POST http://localhost:8888/<tag>?json=<json>                                                                                                                                       
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}                                                                                                                     
# @see http://docs.fluentd.org/articles/in_http                                                                                                                                      
<source>
  type http
  port 8827
</source>

td-agent.log

2016-05-02 09:37:24 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-05-02 09:37:24 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2016-05-02 09:37:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `publish' for nil:NilClass" tag="test"
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sns-2.1.7/lib/fluent/plugin/out_sns.rb:71:in `block in emit'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:54:in `call'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:54:in `each'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sns-2.1.7/lib/fluent/plugin/out_sns.rb:67:in `emit'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event_router.rb:88:in `emit_stream'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event_router.rb:79:in `emit'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:186:in `on_request'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:353:in `call'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:353:in `on_message_complete'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:255:in `<<'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:255:in `on_read'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:123:in `on_readable'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:186:in `on_readable'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run_once'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run'
  2016-05-02 09:37:39 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_http.rb:122:in `run'  <==== Failed
2016-05-02 09:37:50 +0000 debug.test: {"json":"message"}  <==== Successful 

I was able to list the topics on SNS on ec2 as well so no issues with credentials

aws configure

AWS Access Key ID [****************XXXX]: 
AWS Secret Access Key [****************XXXX]: 
Default region name [us-east-1]: 
Default output format [json]: 

aws sns list-topics

{
    "Topics": [
        {
            "TopicArn": "arn:aws:sns:us-east-1:XXX"
        }, 
        {
            "TopicArn": "arn:aws:sns:us-east-1:XXX"
        }
    ]
}

Support also fluentd v0.12 and v0.14

Currently fluentd v0.10 is required, would be very useful if newer fluentd versions were supported.

Feature: Add support for SNS Message Attributes

SNS Message Attributes provides support for sending additional metadata about the message.

It seems that this is only available in the AWS SDK v2, so a migration to the newer SDK is needed (v1 is used ATM).

Emit transaction failed: error_class=NoMethodError error="undefined method `publish' for nil:NilClass" tag="public-api.access"

I'm trying to upload individual log entries as messages to SNS, but I'm getting this error message:
emit transaction failed: error_class=NoMethodError error="undefined method `publish' for nil:NilClass" tag="public-api.access" and a stack trace, whenever an entry to the log is added.

Here's my config:

@type tail path /site/public-api/logs/public.log pos_file /site/public-api/logs/public.log.pos tag public-api.access format json type sns

sns_topic_name XXXXXX
aws_key_id XXXXXXX
aws_sec_key XXXXXXX

sns_endpoint sns.us-west-2.amazonaws.com

Any help is appreciated

Secondary configuration not supported

Configuration:

<match http_source>
  type sns
  sns_topic_name XX
  aws_key_id XX
  aws_sec_key XX
  sns_endpoint XX

  <secondary>
    type file
    path /var/log/td-agent/failed_events_http_source
  </secondary>
</match>

# Accept events from HTTP source at http://SOMEURL:8827/http_source/<JSON EVENT>
<source>
  type http
  port 8827
</source>

Steps to reproduce:

1. Do the above configuration
2. Start HTTP traffic
3. Delete the SNS topic

Expectation:
Events should now be appended to /var/log/td-agent/failed_events_http_source

Outcome:
On console:

500 Internal Server Error
Topic does not exist

Reference: http://docs.fluentd.org/articles/buffer-plugin-overview#secondary-output