jmal98 / sqs-exporter Goto Github PK
View Code? Open in Web Editor NEWA Prometheus Exporter for the Amazon Simple Queue Service (SQS)
License: Apache License 2.0
A Prometheus Exporter for the Amazon Simple Queue Service (SQS)
License: Apache License 2.0
Are there plans to add fix for log4j vulnerability for this exporter
I created a deployment for this sqs-exporter using the container as instructed here. It seems that this exporter not get's discovered from my prometheus. Are there any clues to investigate why?
Thanks, in advance.
FIrst, I really appreciate you putting this repo together! I had some challenges getting the metrics from this container into my k8s based prometheus. I could curl the service and it was working fine but not showing up in prometheus.
After much searching found out you have to annotate your deployments for prometheus to discover them. I'm pasting my simple example below in hopes you can include this type of info in your README.md Perhaps this is common knowledge but my sense is this would help noobs like me get working faster.
Deployment.yaml
kind: Deployment
metadata:
name: sqs-exporter
labels:
app: sqs-exporter
spec:
replicas: 1
selector:
matchLabels:
app: sqs-exporter
template:
metadata:
labels:
app: sqs-exporter
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "9384"
spec:
containers:
- name: sqs-exporter
image: jmal98/sqs-exporter
ports:
- containerPort: 9384
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: batch-iam
key: access-key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: batch-iam
key: secret
service.yaml
apiVersion: v1
kind: Service
metadata:
name: sqs-exporter
labels:
app: sqs-exporter
spec:
selector:
app: sqs-exporter
ports:
- protocol: TCP
port: 9384
targetPort: 9384
Is this project still maintained? Also, is there any way to run the exporter without docker?
Hi, the exporter stopped working, can't see any sqs metrics getting scraped. Is this project still maintained?
I've noticed this after creating a new queue and not updating the exporter IAM role.
We had 4 queues:
qa-queue-cpu
qa-queue-gpu
qa-queue-mem
qa-queue-tasks
The exporter was configured with SQS_QUEUE_NAME_PREFIX=qa
and was working fine, yesterday we added a new queue qa-queue-split
and didn't update the IAM role for the exporter, then I noticed that the qa-queue-tasks
metrics are gone, but all the other are working fine.
After updating the IAM role I finally got metrics from all the queues.
It seems to me that queues are being sorted alphabetically, and if the exporter doesn't have access to one of them, it'll stop at that one, this explains why it didn't get metrics from a queue that was working before we added the new one.
Hi,
I'm trying this export I have a problem on the queue detection.
I want to set manually which queues are monitored.
So I see the exporter support this environment variable: SQS_QUEUE_NAMES
set to myQueue0,MyQueue1
But the exporter will fetch all queues ...
When I remove all list queues capabilities the exporter doesn't return metrics
I don't understand why I have this isssue because when I'm reading the code:
String queueNames = System.getenv("SQS_QUEUE_NAMES");
String queueNamePrefix = System.getenv("SQS_QUEUE_NAME_PREFIX");
if (queueNames != null) {
// find the URLs for the named queues .
String[] names = queueNames.split(",");
queueUrls = new ArrayList<String>();
for(String name : names) {
queueUrls.add(sqs.getQueueUrl(name).getQueueUrl());
}
} else {
// get URLs for all queues visible to this account (with prefix if specified)
ListQueuesResult queues = sqs.listQueues(queueNamePrefix); //If null is passed in the whole unfiltered list is returned
queueUrls = queues.getQueueUrls();
}
It seems the code enter in the else instead of the if or I miss something (I'm not java dev)
version: jmal98/sqs-exporter:0.0.5
iam policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1542622727660",
"Action": [
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ListDeadLetterSourceQueues",
"sqs:ListQueueTags",
"sqs:ListQueues"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
container logs:
19 Nov 2018 16:08:30,109 org.jmal98.metrics.collector.Sqs:44 AmazonSQS client is connected to region: (eu-west-1)
2018-11-19 16:08:31.964:INFO::main: Logging initialized @3733ms to org.eclipse.jetty.util.log.StdErrLog
2018-11-19 16:08:32.051:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_171-b11
2018-11-19 16:08:32.112:INFO:oejs.AbstractConnector:main: Started ServerConnector@3574e198{HTTP/1.1,[http/1.1]}{0.0.0.0:9384}
2018-11-19 16:08:32.114:INFO:oejs.Server:main: Started @3882ms
19 Nov 2018 16:08:32,115 org.jmal98.sqs.exporter.Application:51 Exporter has started.
Today we added our first FIFO sqs queue and we noticed our sqs-exporter stopped working. Apon checking the Prometheus targets (http://127.0.0.1:9090/targets) I could see the sqs-exporter servicemonitor was showing as down
with the error message:
invalid metric type "fifo_sqs_approximatenumberofmessagesdelayed gauge"
As a workaround, we added the SQS_QUEUE_NAMES
ENV var to our sqs-exporter deployment
Manifest to whitelist the queues we want metrics for:
spec:
containers:
- name: sqs-exporter
...
env:
- name: SQS_QUEUE_NAMES
value: "redacted_queue_name1,redacted_queue_name2,redacted_queue_name3"
This way, our new FIFO queue is ignored.
It is my understanding that FIFO queues seem to have to have a name the ends in .fifo
(source). Maybe the .
character in the queue name is causing a issue?
Please close this issue if I'm misunderstanding something - which is very possible.
It seems that the exported metrics are not compatible with older versions of prometheus. We unfortunately are still using version 1.1.2 and it complains about the following error:
text format parsing error in line 123: second HELP line for metric name "sqs_approximatenumberofmessagesdelayed"
I suspect, that it is due to duplicated TYPE
and HELP
hints.
Do you think it is possible to have them only written once?
The exported metrics data looks like this:
# HELP sqs_approximatenumberofmessagesdelayed The approximate number of messages that are waiting to be added to the queue.
# TYPE sqs_approximatenumberofmessagesdelayed gauge
sqs_approximatenumberofmessagesdelayed{queue="queue-1",} 0.0
# HELP sqs_approximatenumberofmessages The approximate number of visible messages in a queue.
# TYPE sqs_approximatenumberofmessages gauge
sqs_approximatenumberofmessages{queue="queue-1",} 0.0
# HELP sqs_approximatenumberofmessagesnotvisible The approximate number of messages that have not timed-out and aren't deleted.
# TYPE sqs_approximatenumberofmessagesnotvisible gauge
sqs_approximatenumberofmessagesnotvisible{queue="queue-1",} 0.0
# HELP sqs_approximatenumberofmessagesdelayed The approximate number of messages that are waiting to be added to the queue.
# TYPE sqs_approximatenumberofmessagesdelayed gauge
sqs_approximatenumberofmessagesdelayed{queue="queue-2",} 0.0
# HELP sqs_approximatenumberofmessages The approximate number of visible messages in a queue.
# TYPE sqs_approximatenumberofmessages gauge
sqs_approximatenumberofmessages{queue="queue-2",} 0.0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.