Comments (13)
I start getting a grasp of what might be happening in here.
ex_aws
claims in their README the default JSON decoder is Poison
:
https://github.com/ex-aws/ex_aws/blob/b40c33e55ace5820d20888ba2a27c5922daba233/README.md#json-codec-configuration
Yet in their defaults they set Jason
(changed 2 years ago):
https://github.com/ex-aws/ex_aws/blob/554465b8783b903ed831e518f463fb2878d6f50e/lib/ex_aws/config/defaults.ex#L10
In mix
, they set Jason
as an optional dependency:
https://github.com/ex-aws/ex_aws/blob/master/mix.exs#L40
When the plugin is built, Jason
is not downloaded (Poison
is instead as it's a dependency of the Plugin) and therefore not included in the release. I will replace Poison
with Jason
and make a fix on ex_aws
.
I still need to verify why it works in Docker but I suspect it's because Jason
is installed in there as native library.
from rabbitmq-cloudwatch-exporter.
Apologies for the disruption, elixir
and logger
dependencies were updated too.
Please let me know if the issue persists.
from rabbitmq-cloudwatch-exporter.
The package has been built using Erlang 24, the error:
This BEAM file was compiled for a later version of the run-time system than 23.
Suggests you are using erlang <= 23. Are you sure your system is running Erlang 24?
from rabbitmq-cloudwatch-exporter.
I noticed it might take some time between AWS receives the metrics and it renders it. I would recommend you to wait half an hour before checking if new metrics are in.
from rabbitmq-cloudwatch-exporter.
Also please note that Debian Erlang packages are a bit challenging to handle as they split all dependencies into separate packages. Moreover, they tend to be always a bit out-of-date.
RabbitMQ recommends different sources for Erlang packages. I usually use those from Erlang Solutions. They package the whole eco-system into a single .deb
which makes it easier to get your environment correctly configured.
from rabbitmq-cloudwatch-exporter.
I've deleted my today's post as I was doing actions on the wrong instance, inviting you to delete yours too to clear out this issue for search purpose, and sorry about that :)
Back to the pb, I did update both dependencies logger and elixir but still having this log :
2021-08-02 13:09:47 =ERROR REPORT====
** Generic server rabbitmq_cloudwatch_exporter terminating
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{queue,[{export_regex,#{'__struct__' => 'Elixir.Regex',opts => <<>>,re_pattern => {re_pattern,0,0,0,<<69,82,67,80,50,1,0,0,0,0,0,0,1,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,131,0,19,29,97,29,115,29,103,29,95,29,109,29,115,29,95,94,12,119,0,49,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,103,29,101,29,110,29,101,29,114,29,97,29,116,29,111,29,114,94,12,119,0,47,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,115,29,101,29,108,29,101,29,99,29,116,29,111,29,114,94,12,119,0,43,29,100,29,108,29,113,29,95,29,115,29,116,29,111,29,114,29,101,29,95,29,112,29,104,29,111,29,116,29,111,29,95,29,105,29,110,29,102,29,111,119,0,21,29,100,29,108,29,113,29,95,29,114,29,97,29,105,29,108,29,115,119,0,59,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,114,29,97,29,105,29,108,29,115,29,59,29,122,29,105,29,112,29,95,29,112,29,104,29,111,29,116,29,111,29,115,29,117,29,98,29,115,29,101,29,116,120,0,238,0>>},re_version => {<<"8.44 2020-02-12">>,little},source => <<"asg_ms_.*|dlq_ms_media_generator.*|dlq_ms_media_selector.*|dlq_store_photo_info|dlq_rails|dlq_ms_rails;zip_photosubset">>}},{export_metrics,[<<"Messages">>]}]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"(...)\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-08-02 13:09:47 =CRASH REPORT====
crasher:
initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1
pid: <0.7554.1>
registered_name: []
exception exit: {{{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"I(...)g==\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.1163.0>]
message_queue_len: 0
messages: []
links: [<0.7553.1>]
dictionary: []
trap_exit: false
status: running
heap_size: 4185
stack_size: 29
reductions: 216521
neighbours:
neighbour: [{pid,<0.7553.1>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.1163.0>]},{message_queue_len,0},{links,[<0.1164.0>,<0.7554.1>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,12},{reductions,85},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}]
2021-08-02 13:09:47 =SUPERVISOR REPORT====
Supervisor: {local,'Elixir.Singleton.Supervisor'}
Context: child_terminated
Reason: {{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"I(...)==\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}}
Offender: [{pid,<0.7553.1>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{significant,false},{shutdown,5000},{child_type,worker}]
from rabbitmq-cloudwatch-exporter.
Please make sure to download all packages anew as some might have been re-compiled and replaced.
Here is how I tested the plugin with a Dockerfile.
FROM rabbitmq:3.8-management
ARG PLUGIN_DIR=plugins/
ARG PLUGIN_URL=https://api.github.com/repos/noxdafox/rabbitmq-cloudwatch-exporter/releases
# Install essential packages
RUN apt update && apt install -y curl jq wget
# Dowload plugin and dependencies
RUN for url in $(curl $PLUGIN_URL | jq .[0].assets[].browser_download_url -r); \
do wget $url -P $PLUGIN_DIR; done
# Enable plugin
RUN rabbitmq-plugins enable --offline rabbitmq_cloudwatch_exporter
# Configure the broker to use the plugin
RUN echo 'loopback_users.guest = false \n\
cloudwatch_exporter.aws.region = "YOUR_AWS_REGION" \n\
cloudwatch_exporter.aws.access_key_id = "YOUR_KEY_ID" \n\
cloudwatch_exporter.aws.secret_access_key = "YOUR_SECRET_KEY" \n\
cloudwatch_exporter.namespace = "RabbitMQPluginTest" \n\
cloudwatch_exporter.export_period = 20 \n\
cloudwatch_exporter.metrics.overview.enable = true' \
>> /etc/rabbitmq/rabbitmq.conf
$ docker build . -t rmq-docker
$ docker run -it -p 15672:15672 rmq-docker:latest
Browse to http://localhost:15672
login with default credential. I created few queues and published few messages. I could see the metrics in AWS CW. No error logs, just a warning:
Problem reading some plugins: [{"/opt/rabbitmq/plugins/lager-3.8.0.ez",
duplicate_plugin}]
I cannot remove lager
as a package as in some old deployment it was giving me some issues.
from rabbitmq-cloudwatch-exporter.
Hi guys, tomorrow I'm going to try this plugin with RabbitMQ 3.9.1 and Erlang 24 in an EC2 instance.
Today I successfully installed this plugin retrieving packages of the latest version (1.0.0) via GitHub API, in a Docker container (RabbitMQ 3.9.1 and Erlang 24)... without trying metric creation in CloudWatch, though 😅.
Maybe useful, GitHub API call:
curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/noxdafox/rabbitmq-cloudwatch-exporter/releases | jq ".[0].assets[].browser_download_url"
from rabbitmq-cloudwatch-exporter.
I've deleted my today's post as I was doing actions on the wrong instance, inviting you to delete yours too to clear out this issue for search purpose, and sorry about that :)
Back to the pb, I did update both dependencies logger and elixir but still having this log :2021-08-02 13:09:47 =ERROR REPORT==== ** Generic server rabbitmq_cloudwatch_exporter terminating ** Last message in was export_metrics ** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{queue,[{export_regex,#{'__struct__' => 'Elixir.Regex',opts => <<>>,re_pattern => {re_pattern,0,0,0,<<69,82,67,80,50,1,0,0,0,0,0,0,1,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,131,0,19,29,97,29,115,29,103,29,95,29,109,29,115,29,95,94,12,119,0,49,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,103,29,101,29,110,29,101,29,114,29,97,29,116,29,111,29,114,94,12,119,0,47,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,115,29,101,29,108,29,101,29,99,29,116,29,111,29,114,94,12,119,0,43,29,100,29,108,29,113,29,95,29,115,29,116,29,111,29,114,29,101,29,95,29,112,29,104,29,111,29,116,29,111,29,95,29,105,29,110,29,102,29,111,119,0,21,29,100,29,108,29,113,29,95,29,114,29,97,29,105,29,108,29,115,119,0,59,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,114,29,97,29,105,29,108,29,115,29,59,29,122,29,105,29,112,29,95,29,112,29,104,29,111,29,116,29,111,29,115,29,117,29,98,29,115,29,101,29,116,120,0,238,0>>},re_version => {<<"8.44 2020-02-12">>,little},source => <<"asg_ms_.*|dlq_ms_media_generator.*|dlq_ms_media_selector.*|dlq_store_photo_info|dlq_rails|dlq_ms_rails;zip_photosubset">>}},{export_metrics,[<<"Messages">>]}]}]}] ** Reason for termination == ** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"(...)\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]} 2021-08-02 13:09:47 =CRASH REPORT==== crasher: initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1 pid: <0.7554.1> registered_name: [] exception exit: {{{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"I(...)g==\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]} ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.1163.0>] message_queue_len: 0 messages: [] links: [<0.7553.1>] dictionary: [] trap_exit: false status: running heap_size: 4185 stack_size: 29 reductions: 216521 neighbours: neighbour: [{pid,<0.7553.1>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.1163.0>]},{message_queue_len,0},{links,[<0.1164.0>,<0.7554.1>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,12},{reductions,85},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}] 2021-08-02 13:09:47 =SUPERVISOR REPORT==== Supervisor: {local,'Elixir.Singleton.Supervisor'} Context: child_terminated Reason: {{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"(...)\",\n \"SecretAccessKey\" : \"(...)\",\n \"Token\" : \"I(...)==\",\n \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}} Offender: [{pid,<0.7553.1>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{significant,false},{shutdown,5000},{child_type,worker}]
@noxdafox @falcocoris same issue for me and no metric on CloudWatch. EC2 instance with Ubuntu18.04, RabbitMQ 3.9.1 and Erlang 24.0.5.
Asap I'm going to try installing the RabbitMQ 3.8.X.
from rabbitmq-cloudwatch-exporter.
This is the error with RabbitMQ 3.8.19
** Client <0.1018.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,233}]},{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1021}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/conf
ig.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream'
,do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/con
fig.ex"},{line,79}]}]
2021-08-03 18:19:07.659 [error] <0.1016.0> CRASH REPORT Process 'Elixir.ExAws.Config.AuthCache' with 0 neighbours crashed with reason: call to undefined function 'Elixir.Jason':'decode!'(<<"
{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n \"Type\" : \"AWS-HMAC\",\n \"Acc...">>)
2021-08-03 18:19:07.659 [error] <0.1018.0> ** Generic server rabbitmq_cloudwatch_exporter terminating
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{exchange,[{export_metrics,[<<"PublishIn">>,<<"PublishOut">>]}]}
,{node,[]},{overview,[]},{queue,[{export_metrics,[<<"Messages">>,<<"MessagesReady">>]}]},{vhost,[]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"...
\",\n \"SecretAccessKey\" : \"...\",\n \"Token\" : \"...\",\
n \"Expiration\" : \"2021-08-03T22:17:42Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache'
,refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,tr
y_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixi
r.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_cli
ent => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_bac
koff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line
,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerabl
e.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'E
lixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-08-03 18:19:07.660 [error] <0.1018.0> CRASH REPORT Process <0.1018.0> with 1 neighbours exited with reason: {{undef,[{'Elixir.Jason','decode!',[<<"{\n \"Code\" : \"Success\",\n \"Last
Updated\" : \"2021-08-03T16:11:24Z\",\n \"Type\" : \"AWS-HMAC\",\n \"AccessKeyId\" : \"...\",\n \"SecretAccessKey\" : \"....\",\n \"T
oken\" : \"....">>],...},...]},...} in 'Elixir.GenServer':call/3 line 1024
2021-08-03 18:19:07.660 [error] <0.554.0> Supervisor 'Elixir.ExAws.Supervisor' had child 'Elixir.ExAws.Config.AuthCache' started with 'Elixir.ExAws.Config.AuthCache':start_link([]) at <0.101
6.0> exit with reason call to undefined function 'Elixir.Jason':'decode!'(<<"{\n \"Code\" : \"Success\",\n \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n \"Type\" : \"AWS-HMAC\",\n \"Acc.
..">>) in context child_terminated
from rabbitmq-cloudwatch-exporter.
Release updated, Poison
replaced with Jason
. Tested locally and on a Docker container and it seems to be working.
Could you please confirm the fix?
from rabbitmq-cloudwatch-exporter.
I confirm the fix !
Thx for your work mate, I'll close this up :)
from rabbitmq-cloudwatch-exporter.
Yes, I confirm too! Both RabbitMQ versions (3.8.x and 3.9.1) work! Thanks @noxdafox
from rabbitmq-cloudwatch-exporter.
Related Issues (20)
- Error: dependent plugins [goldrush] not found; used by [lager]. HOT 6
- Update plugin configuration in runtime (without a restart) HOT 2
- Metrics not reset (accumulating) HOT 4
- Question: I see that used old style config for node exporter plugin HOT 1
- Error during enable plugin for erlang 21 HOT 3
- fatal: repository 'https://github.com/rabbitmq/rabbitmq-stream-management.git/' not found HOT 3
- Lager failed to install handler lager_cloudwatch_backend into lager_event HOT 8
- getting 'module could not be loaded' string_compat,to_lower in the latest rabbitmq 3.8
- Support for IMDSV2 HOT 5
- Call to undefined function crypto:hmac HOT 6
- Duplicate and dependency errors HOT 4
- Missing dependencies on enabling plugin HOT 5
- CloudWatch routingKey Metrics HOT 3
- cloudwatch exporter - exporting logs issue HOT 5
- No logs published to Cloudwatch HOT 16
- Metrics export not working in ME-south-1 HOT 9
- Logs are not being sent to Cloudwatch
- a
- bug with cluster name / node IP metric aggregation HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-cloudwatch-exporter.