oliviabarrick/fluxcloud

Slack notifications for Weave Flux without Weave Cloud

Fluxcloud is a tool to receive events from the Weave flux.

Weave Flux is a useful tool for managing the state of your Kubernetes cluster.

Fluxcloud is a valid upstream for Weave, allowing you to send Flux events to Slack or a webhook without using Weave Cloud.

Setup

Please see the Weave Flux setup documentation for setting up Flux.

To use Fluxcloud, you can deploy fluxcloud as either a sidecar to Flux or a seperate deployment.

To deploy as a sidecar, see examples/flux-deployment-sidecar.yaml. To deploy independently, see examples/fluxcloud.yaml.

Set the following environment variables in your chosen deployment:

• SLACK_URL: the Slack webhook URL to use.
• SLACK_USERNAME: the Slack username to use when sending messages.
• SLACK_TOKEN (optional): legacy Slack API token to use.
• SLACK_CHANNEL: the Slack channel to send messages to.
• SLACK_ICON_EMOJI: the Slack emoji to use as the icon.
• MSTEAMS_URL: the Microsoft Teams webhook URL to use
• GITHUB_URL: the URL to the Github repository that Flux uses, used for Slack links.
• WEBHOOK_URL: if the exporter is "webhook", then the URL to use for the webhook.
• EXPORTER_TYPE (optional): The types of exporter to use in comma delimited form. (Ex: slack,webhook) (Choices: slack, msteams, webhook, Default: slack)
• JAEGER_ENDPOINT (optional): endpoint to report Jaeger traces to.

And then apply the configuration:

kubectl apply -f examples/fluxcloud.yaml

Set the --connect flag on Flux to --connect=ws://fluxcloud.

Exporters

There are multiple exporters that you can use with fluxcloud. If there is not a suitable one already, feel free to contribute one by implementing the exporter interface!

Slack

The default exporter to use is Slack. To use the Slack exporter, set the SLACK_URL, SLACK_USERNAME, and SLACK_CHANNEL environment variables to use. You can also optionally set the EXPORTER_TYPE to "slack".

Sending notifications to multiple channels

If sending notifications to only one channel is unsufficient for your use case you can configure fluxcloud to send them to multiple channels based upon the namespace(s) from the created and/or updated resources. This is done by setting a comma separated <channel>=<namespace> string as the SLACK_CHANNEL environment variable.

If you for example want to send notifications of all events to #k8s-events but only events from namespace team-b to #teamb you would set the following string: SLACK_CHANNEL=#k8s-events=*,#team-b=team-b.

Microsoft Teams

Set the environment variable MSTEAMS_URL to the URL generated on activation of an Incoming Webhook in a Microsoft Teams channel.

Webhooks

Events can be sent to an arbitrary webhook by setting the EXPORTER_TYPE to "webhook" and then setting the WEBHOOK_URL to the URL to send the webhook to.

Fluxcloud will send a POST request to the provided URL with the encoded event as the payload.

Formatting commit links

By default, commit links are formatted for Github. It is possible to format them for another VCS system, such as Bitbucket, by overriding the commit template.

The commit template is a go template that supports two variables:

• VCSLink: which is the GITHUB_URL configuration option.
• Commit: which is the commit id.

The default is:

{{ .VCSLink }}/commit/{{ .Commit }}

For example, to override to work for Bitbucket, set the COMMIT_TEMPLATE environment variable to:

{{ .VCSLink }}/commits/{{ .Commit }}

Versioning

Fluxcloud follows semver for versioning, but also publishes development images tagged with $BRANCH-$COMMIT.

To track release images:

fluxctl policy -c kube-system:deployment/fluxcloud --tag-all='v0*'

To track the latest pre-release images:

fluxctl policy -c kube-system:deployment/fluxcloud --tag-all='master-*'

And then you can automate it:

fluxctl automate -c kube-system:deployment/fluxcloud

Build

To build fluxcloud, you can either use go:

go build -o fluxcloud ./cmd/

Or, to run a full CI build, download hone:

hone

fluxcloud's Issues

[ANN] - We're continuing the development of fluxcloud in a new repo

Hi everyone,

Since we've been working on this project a lot recently and after not receiving response from the maintainer (not even through email) for a long time, we've decided to continue the development in our fork and allow external contributors to submit their PR's.

We're more than happy to review your submissions and continue to grow this project there.

The new repo is https://github.com/topfreegames/fluxcloud

Cheers!

3031 taken

The new default install of flux recommends this line:

        # serve /metrics endpoint at different port.
        # make sure to set prometheus' annotation to scrape the port value.
        - --listen-metrics=:3031

In order to get fluxcloud to start I needed to change their prometheus port, since I didn't see any way to specify a different port in the fluxcloud env vars.

Not sure what the best answer is here, at least mentioning it in the deployment file would be helpful. If you could change to an unused port that would be great.

Possibility of using more than one exporter at once.

We would like to both trigger a webhook and a slack message. Is it currently possible to configure and use two different exporters at the same time?

If not, could we get this in as a feature request?

Can we have a new release?

Thanks for fluxcloud!

Is it possible to have a new tagged release - looks like the multiple-exporter option is not available in a release (and it's a really cool feature!).

Thanks again.

Slack username not included in message

With default config of fluxcloud, and the username env var set to something other than Flux, I can't see the username being changed in Slack channel. My env config:

- name: SLACK_USERNAME
  value: "Cool cluster"
- name: SLACK_URL
  value: "https://hooks.slack.com/services/XXXXXXXXX"
- name: SLACK_CHANNEL
  value: "#my-channel"
- name: SLACK_ICON_EMOJI
  value: ":sync:"
- name: GITHUB_URL
  value: "https://github.com/my-repo"
- name: LISTEN_ADDRESS
  value: ":3032"

support discord webhooks

Would you please consider adding proper support for discord webhooks to fluxcloud?

Although you can configure fluxcloud to use a discord webhook using the /slack emulation trick (eg, https://discordapp.com/api/webhooks/1234567890/abc123efg456/slack) and it works for smaller flux repo changes, fluxcloud will choke sending very large payloads.

Here's the error I get when fluxcloud is sent a giant payload from flux:

Could not post to slack, status: 400 Bad Request
Exporter Slack got an error: Could not post to slack, status: 400

I assume that discord limits webhook payload sizes to a smaller size than slack or something like that. Either way, the /slack endpoint is not a reliable way to receive fluxcloud messages on discord at the moment.

Thanks!

Communication issues between Weave Flux and Fluxcloud

Hey ! 🚀

Describe the bug

• Weave Flux cannot connect to Fluxcloud (connection refused)
• Fluxcloud has handshake issues when not finding a token inside the Connection header...

To Reproduce

1. Install Weave Flux (helm operator is active) + uncomment --connect: ws
2. Create Mattermost incomming Webhook
3. Install Fluxcloud using examples/fluxcloud.yaml
4. Wait for events to appear on Mattermost... Die of old age...

As the connection from Weave Flux to Fluxcloud seems buggy, there is no logs posted anywhere.

Expected behavior
Connection from Weave Flux to Fluxcloud should be working and logs should be posted to the webhook.

Logs
Logs of fluxcloud:

Using Webhook exporter
Request for:/v11/daemon
client connected!
Request for:/
upgrade:websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
Request for:/
upgrade:websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
Request for:/
upgrade:websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
Request for:/
upgrade:websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header

Logs of Weave Flux:

...
caller=upstream.go:118 component=upstream err="executing websocket ws://fluxcloud/v11/daemon: dial tcp 10.233.55.12:80: connect: connection refused"
...

examples/fluxcloud.yaml

    spec:
      securityContext:
        runAsUser: 999
      containers:
      - name: fluxcloud
        image: justinbarrick/fluxcloud:v0.3.9
        imagePullPolicy: Always
        ports:
        - containerPort: 3032
        env:
        - name: EXPORTER_TYPE
          value: webhook
        - name: WEBHOOK_URL
          value: http://10.10.2.100:8080/hooks/cbzjhjthxbyzzqmq1bgj1i5f8o
        - name: GITHUB_URL
          value: http://10.10.2.100:10080/INFRA/projects.git
        - name: LISTEN_ADDRESS
          value: ":3032"

Additional context

• Kubernetes: 1.14.3
• Fluxcloud: justinbarrick/fluxcloud:v0.3.9
• Weave Flux: fluxcd/flux:1.15.0

Debugging with helm flux install

Hi,

I've installed flux using helm and then fluxcloud using examples/fluxcloud.yaml. Both are on the same namespace:

I added

- --connect=ws://fluxcloud

I can see the two are connected from fluxcloud logs:

Using Slack exporter
Request for:/v10/daemon
client connected!

However I'm not receiving any notifications to Slack. Any ideas how I can debug this further?

Expose a HTTP endpoint with the last $N events for polling

I was wondering if it would be desirable for fluxcloud to expose a HTTP endpoint which would return a list of the events it's received (limited to some user configurable value for memory reasons) that could be polled by an external service?

We have a use case where we'd like to receive flux events in a system where we can't accept inbound connections (so webhooks don't work) but we can make outbound connections to other services.

I'm not sure if this even makes sense for fluxcloud or should be its own service that fluxcloud can talk to via webhooks. Maybe even something like a redis/kafka/whatever exporter would work (fluxcloud writes events to redis, we poll redis for new events 🤷‍♂️ ).

Wanted to find out what you think before trying to implement this.

Only for Public github?

Hi.
I've deployed both deployment and sidecar.. tried both.
It does not work. I've created a webhook and I see nothing in slack.
don't see any errors from fluxcloud or anything.

just nothing is happening.
tried this on two different clusters.

EKS cluster.
Kubernetes 1.13

flux is working.

GITHUB_URL does not work for BitBucket :upside_down_face:

Well, duh, but there's one simple difference:

GitHub has /commit/<sha>
BitBucket has /commits/<sha>.

There's only a single-character between the two URL schemes.

Could URL support please be extended to include BitBucket as well?

adding parameter in commitTemplate for /commit/

I am trying to use it with bitbucket and the path in bitbucket it is commits. Would it be good to take the whole commit link in param and append commmit id or template to replace the commit number?

Support forwarding events to multiple upstreams

It would be useful to forward events to multiple upstreams, for example, you might want to get notifications on slack, send them to the fluxcloud-webhook-monitor, and Sentry.

Invalid service ID

My connection between flux and fluxcloud works fine but I am experiencing something similar to this issue #26
I'm using flux 1.19.0 and fluxcloud 0.3.9
here is the log from fluxcloud:

Request for:/v6/events
{"id":0,"serviceIDs":["\u003ccluster\u003e:namespace/"],"type":"sync","startedAt":"2020-04-15T10:34:18.964351567Z","endedAt":"2020-04-15T10:34:18.964351567Z","logLevel":"info","metadata":{"commits":[{"revision":"7eda317cbeaa70fb1b76f73f977a71725e7c2f21","message":"test namespace"}],"includes":{"other":true},"errors":[{"ID":"\u003ccluster\u003e:namespace/","Path":"namespaces/test.yaml","Error":"running kubectl: error: error validating \"STDIN\": error validating data: ValidationError(Namespace.metadata): unknown field \"names\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false"}]}}
parsing <cluster>:namespace/: invalid service ID
Request for:/v6/events

parsing <cluster>:namespace/: invalid service ID - everytime this occurs in the log, message is not sent. It only happens when there is some kind of error on flux side, e.g. yaml resource validation. So for now we are not notified about errors which is not good.

Issue with cluster-level resources

I've been seeing this issue a lot lately:

ts=2019-03-19T18:15:52.144930335Z caller=loop.go:412 component=sync-loop err="executing HTTP request: 400 Bad Request parsing <cluster>:clusterrole/kube2iam: invalid service ID\n"

It causes fluxctl sync to return Error: timeout, but no other discernible issues.

See fluxcd/flux#1843

Implement shoutrrr to support different types of notification options

https://github.com/containrrr/shoutrrr currently supports:

• gotify
• hangouts
• ifttt
• logger
• mattermost
• pushbullet
• pushover
• slack
• smtp
• standard
• teams
• telegram

This could offload a lot code to this dependency

README build status badge broken

Build status badge in readme does not show anything.

COMMIT_TEMPLATE env var not used

When trying to use fluxcloud notifications with a private bitbucket-based repository, the COMMIT_TEMPLATE does not seem to do anything.

From the kubectl describe pod command:

Environment:
  ...
  COMMIT_TEMPLATE:   {{ .VCSLink }}/commits/{{ .Commit }}

Resulting link in the notification (with placeholder words for the domain and project/repo): http://git.private.com/projects/myproject/repos/my-flux-repo/commit/sha

Edit: not sure if I should create a separate issue for that, but when I change the chart in my flux repo, no notification is sent, though the event can be seen in the fluxcloud logs, is that an intentional behaviour?

Re-deployment notification after manual delete

If I delete an helmrelease object manualy and it's get redeployed automatically,
I won't get any fluxcloud notification.
I would expect a notification anytime a deployment action was done.

Implement 'failure' notifications

It is great to be able to track flux events as they happen, however the really useful side of notifications are those that bring ones attention to when things don't go according to plan.

Positive notifications create a lot of chatter and are quickly being ignored by developer teams.
In most real-life scenarios it is typical to only enable notifications for when exceptions, such as failed deployments, happen.

Google Chat exporter

@justinbarrick are you interested in receiving a PR for a Google Chat exporter?

Messages can be formatted as a card or simple text.

Seems like it will work similar to others as all it needs is the right message format and a URL. Creating Google Chat webhooks is relatively simple.

Issue with CRDs

I deployed bitnami's sealed secrets (https://github.com/bitnami-labs/sealed-secrets) into my cluster and now I get the following error in fluxcloud.

Request for:/v6/events
parsing default:customresourcedefinition/sealedsecrets.bitnami.com: invalid service ID
Request for:/v6/events
parsing default:customresourcedefinition/sealedsecrets.bitnami.com: invalid service ID
Request for:/v6/events
parsing default:customresourcedefinition/sealedsecrets.bitnami.com: invalid service ID
Request for:/v6/events
parsing default:customresourcedefinition/sealedsecrets.bitnami.com: invalid service ID

Append custom text to title

I'm using MicrosoftTeams and unless I create a different connector with different names for each of my clusters, I have no way to identify which cluster produced the message.

My suggestion is add the ability to append custom prefix to the title for such use cases. WDYT?

Automatically test and bump flux library versions when new flux releases are published

Git Provider Deployment Status

Getting simple feedback whether or not a commit has been applied by Flux is great for developers. Slack notifications help but can be a bit much. A minimalist option would be to update the status of the given commit.

Many git providers allow external sources to set the status of a commit.

• GitHub
• Gitlab
• Bitbucket

This feature has been suggested to be added to Flux in fluxcd/flux#2899 but is considered a bit to vendor specific to be added to the project.

Is this something that could be added to fluxcloud instead? I haven't looked to much at the codebase but I don't think I would have any problems implementing the feature if it would be accepted.

Support for additional exporters

Hey, we'd like to add support for an additional exporter (Kafka or Redis, we're not sure yet honestly) to support some different use cases we have at my workplace. We're happy to do the work to add this functionality but I wanted to check if that was a desirable feature for this project before starting to write any code?

Thanks.

Questions about Project!

Hello,

I was pointed to this project when I asked if flux would have alerts for deploys etc at OSS.

So i have a few questions!

Is there any reason why not to implement that direct at flux oss?

This works with helm-op ? We use flux with helm-op for everything.
If yes, does it get git url from the release.

Are there other goals? (cause of the name)

Is there an example msg on how it would appears?

Sorry I didn't look at the code yet, so I am asking this questions!

Thanks!

Log about a failed deployment

I would like to have the information from helm-operator sent via an exporter as well.
In case there is a failed helm deployment, such an information would help us really much.
Are you thinking about implementing such behaviour?

##flux log
ts=2019-06-04T11:26:46.976670041Z caller=sync.go:470 component=cluster method=Sync cmd=apply args= count=9
ts=2019-06-04T11:26:47.299043655Z caller=sync.go:536 component=cluster method=Sync cmd="kubectl apply -f -" took=322.205024ms err=null output="helmrelease.flux.weave.works/deployment-test-container-crash configured"
ts=2019-06-04T11:26:47.314261782Z caller=daemon.go:627 component=daemon event="Sync: b182e3d, default:helmrelease/deployment-test-container-crash" logupstream=true
ts=2019-06-04T11:26:47.747839441Z caller=loop.go:441 component=sync-loop tag=flux-sync old=ea0fd97fb37a55f3da6ccdac6adc22c8fbf81eb7 new=b182e3dc8f2b9afb1d78a75b008265a1207de239
ts=2019-06-04T11:26:47.923236797Z caller=loop.go:103 component=sync-loop event=refreshed url=###HIDDEN_URL### branch=master HEAD=b182e3dc8f2b9afb1d78a75b008265a1207de239

##fluxcloud registered events
`Request for:/v6/events
{"id":0,"serviceIDs":["default:helmrelease/deployment-test-container-crash"],"type":"sync","startedAt":"2019-06-04T11:26:28.8560952Z","endedAt":"2019-06-04T11:26:28.8560952Z","logLevel":"info","metadata":{"commits":[{"revision":"b182e3dc8f2b9afb1d78a75b008265a1207de239","message":"test failed deployment event"}],"includes":{"other":true}}}
{"channel":"#weave-flux-notif","icon_emoji":":ghost:","username":"Flux Operator","attachments":[{"color":"#4286f4","title":"Flux deployment:","title_link":"###HIDDEN_URL###","text":"Event: Sync: b182e3d, default:helmrelease/deployment-test-container-crash\nCommits:\n\n* \u003c###HIDDEN_URL###|b182e3\u003e: test failed deployment event\n\nResources updated:\n\n* default:helmrelease/deployment-test-container-crash"}]}
{#weave-flux-notif 👻 Flux Operator [{#4286f4 ###HIDDEN_MSG&URL### Event: Sync: b182e3d, default:helmrelease/deployment-test-container-crash
Commits:

• <###HIDDEN_URL###|b182e3>: test failed deployment event

Resources updated:

• default:helmrelease/deployment-test-container-crash}]}

##helm-operator logs
ts=2019-06-04T11:27:32.528227847Z caller=chartsync.go:300 component=chartsync warning="unable to proceed with release" resource=default:helmrelease/deployment-test-container-crash release=deployment-test-container-crash err="release requires a rollback before it can be upgraded (FAILED)"

README.md incorrect link to Weave Flux setup

"Please see the Weave Flux setup documentation for setting up Flux."

Tha page shows HTTP 404 Page not found.

Add support for TCP output exporter

I'll submit a PR soon (it's nearly working just need to write tests).

What this effectively means is that we can dump the flux-events via flux-cloud to something like filebeat, and then for example, onto ElasticSearch.

Could not post to MS Teams sometimes

I installed fluxcloud as sidecar and only enable msteam exporter.

For the first time the pod is created, it always failed to send message, the error message is:
"Could not post to MS Teams: context canceled."

However, when I delete the pod and re-created by replicaset, the problem is gone and the message can be sent successfully.

Only send message for certain events

Thanks for this module. When trying it, we saw that there are 3 events reported when flux does a deployment. Is there a way to specify for which of these events a message should be send?

I have looked through the code and found that it would probably not be hard to implement as an additional step in the v6 route. Would you accept such PR?

Unable to send messages to slack/msteams

Installed Flux and deployed examples/fluxcloud.yaml as separate deployment. Communication not happened and I didnt see any logs as well.

Logs:
[{#kubernates *}]
Using Slack exporter

Only send a notification when there is an error

First thing, thanks for this module!

We would be interested in receiving notifications only when there is an error during the flux sync (to reduce notification noise - if there is no error, then we can be fairly sure that eventually changes will be applied).

I'm aware of #12 but I don't think we can use it for this purpose.

Would you be interested in supporting such a feature? I make no promises that I'll have the time to provide a PR, but maybe one day if nobody beats me to it...

report errors?

I love seeing things get deployed, etc. on the slack channel. Would be great if I could get error messages from the flux logs as well, to know when I've committed something wrong.

Add project license

Hey there, we're interested in using and contributing to fluxcloud a bit but before that we'd really like to know what the license of it looks like. Thank you :)

Slack message only shows latest resource in manifest

Currently I have created following manifest -

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    iam.amazonaws.com/permitted: ".*"
  labels:
    name: rbac-testing
  name: rbac-testing
---
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    iam.amazonaws.com/permitted: ".*"
  labels:
    name: rbac-testing-ro
  name: rbac-testing-ro
---
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    iam.amazonaws.com/permitted: ".*"
  labels:
    name: rbac-testing-misc
  name: rbac-testing-misc

the problem is, however, that in the Slack group I can only see the latest resource whereas all the other resources were added as well.

Resources updated:
* <cluster>:namespace/rbac-testing-misc

This could be either a feature or a bug, not sure.

Thanks!