argoproj-labs/argocd-bot

Bot to automate Kubernetes deployment via Github PRs

Argocd Bot

A bot to help automate argo-cd changes via Github PRs.
Currently supports running diffs on open Pull Requests, check the Workflow section for more, or comment argo help on an open PR.

Benefits

Easier Deployments/Fewer Mistakes

• Comment argo diff on an open PR, to view diff between local branch and Kubernetes cluster.
• PR diffs can be easily reviewed by everyone.
• Catch errors in the output of argo diff before applying changes.
• Comment argo sync [app name] to deploy changes, and merge PR.

Lock-down Deployments

• Users can submit changes via Github (after PR approval), without needing cluster credentials.
• Audit changes made to clusters via Github PRs/server logs.

Workflow

This section describes the workflow supported by the bot.

Workflow basics

• User opens a PR in a Kubernetes repo with changes to deployment files.
• User comments with argo diff on the PR.
• Bot checks out current state of PR and runs argocd diff --local. Diff output is posted on the PR as a comment.
• Team can review code changes in the PR, and double check the posted diff.
• Author can iterate, making changes on the PR and re-comment with argo diff once they are ready.
• Deploying Changes:
  • If Auto Sync is enabled: once the PR is merged, ArgoCD server syncs it with production.
  • Otherwise, user can comment argo sync [app name] to sync changes from branch, before merging PR.

PR Example

Locking

When any command is run on by a user, the PR holds a lock, until the it is merged, or unlock is run. The PR lock prevents other users from attempting to run commands on their PRs. This is to synchronize changes on master.
i.e to prevent a scenario like this:

• Alice creates a PR, comments diff, and she's happy with the output posted.
• Bob creates a second PR, and comments diff.
• Bob merges his PR to master.
• Alice's diff output on her PR is now invalid and she might have no idea. Her merge to master, might produce a different state than what her diff had shown.

With locking in place, Bob will not be able to merge his PR until he coordinates with Alice, by either running unlock, or waiting for her PR to get merged first.

Workflow Commands

These are commands that are supported by the bot.

• argo diff: this checks out the current state of the PR, queries the argoCD server at /api/v1/applications and diffs all applications with their current state from the PR.
• argo unlock: this unlocks the current PR, so other users can run diff; see locking section above.
• argo sync [app name]: this syncs changes on current branch
• argo rollback: this rolls back latest change

More commands might be added, run argo help on a PR, to view all supported commands.

Deployment

Create a Github App

Create a new GitHub App here.

• Webhook URL, is the host where the bot will run.
• Webhook Secret, is an optional secret, make sure it matches the config (see below section)
• Private key, generate a new key and place it in the root directory, and update config below.
• Check the generated APP_ID by Github. For more on creating Github apps see

Required Permissions for Github App

Please give the argo-cd app the following permissions:

Read access to administration and metadata
Read and write access to commit statuses, issues, and pull requests 

Update Config

There is an .env_example file that should be renamed to .env. NodeJS will read that file and expose the variables to the bot, when running locally.
When running in Kubernetes, there is a helper script to create k8s secrets from that file (more on this in the kubernetes deployment section).
Here is a description of each parameter:

• PORT is the port that bot will listen on via HTTP.
• LOG_LEVEL can be set to trace, debug, info, warn, error, or fatal.
• KUBECTL_EXTERNAL_DIFF this is used by argocd diff, we pass a helper script to pretti-fy diffs posted on the PR.
• APP_ID is the app id corresponding to the Github app (this is generated on app creation).
• GHE_HOST for Github enterprise installations, specify the hostname. Otherwise leave blank, bot will use Github.com
• GITHUB_REPO this is the repo that the bot will operate on.
• GITHUB_TOKEN generate a Github token for the bot, and give it no scopes. This is just used to clone the repo.
• WEBHOOK_SECRET is the secret configured when creating the Github app (can be left empty if no secret is specified).
• PRIVATE_KEY_PATH is the path to the private key generated for the Github app, this is usually a .pem file.
• ARGOCD_SERVER, this is the ip address/hostname of the argocd server.
• ARGOCD_AUTH_TOKEN it is recommended to generate an automation token using the /api/v1/projects/{project}/roles/{role}/token API. For more information see

Kubernetes Deployment

Docker images of argocd-bot are built here, they are provided as part of releases here

Check the config section above, once you have a .env file that's populated with the correct values run ./helper_scripts/create_kubectl_secrets.sh.
This will generated a k8s secret argocd-bot-secret used by the deployment.

Build manifests using kustomize: npm run manifests

Create deployment from manifests: kubectl create -f deployment/install.yaml

Manual Deployment

See docs here

Development/Contributing

See docs here

argocd-bot's Issues

Useful Features

List of useful features to have

• argo preview to create a preview environment from PR and tear-down environment once PR closes, see argoproj/argo-cd#1157
• post github hook status
• when running argo sync app have an option to wait for the operation to complete
• don't attempt to implement rollback. just let users rollback to whatever version they want, using hstory command, need to think more about how rollbacks should work
• get clone URL from hook, instead of relying on config
• download argocd cli at runtime
• support downloading custom argocd version from a specific URL
• look into if we can integrate with github deployment API

Important Features

Some of the important features we need:

• argo sync to sync all the diffs found on the branch, feels a bit dangerous to me
• fix argo rollback
• argo list, to list all apps tracked in repo
• print message about locking
• timeout for sync
• some config on requiring approval on a PR before sync-ing

Can this work without using a GitHub App?

We run an internal GitHub Enterprise instance. We have github build users for which we can easily generate tokens.

It's not as simple for us to use GitHub Apps. Can this be configured entirely using PAT rather than requiring a GitHub App? The README.md only mentions the GitHub App workflow.

Comments on a closed PR should not hold lock

When a PR is closed/merged and someone posts a comment the bot should just ignore it.

Current behavior is if the PR is closed, the bot responds to the command and creates a lock

Support GitLab PRs

It would be awesome if the argocd-bot would also support GitLab PRs. But it seems this can not be easily achieved due the usage of GitHub-specific probot library.

I am opening the issue just for tracking how many would like this feature. Similar feature requests:

• #10
• #30

improved support for diff command

Should support
diff --auto-sync
and diff --all

tests stub on exec are not working

The JS code execs a few helper bash scripts. I'm using child_process.exec. When trying to mock this out using sinon.stub, the stub is not working (no data is being returned). Seems that the JS code is not using the mock.

const child_process = require('child_process')
const execStub = sinon.stub(child_process, 'exec')
execStub.returns({'stdout': 'test'})

One bot per repo?

We are running Github Enterprise and we have all of our k8s repos within a single Github organization. From the looks of the config file argo-bot is configured one bot per repo? Is there currently support to have argo-bot be available to support all repos within an org?

Code re-factoring checklist

Some useful code refactoring:

• put all constants, like messages and such in a separate JS class/file

Support GitHub App instead of GitHub PAT

WHY

Because GitHub App has full-grained access control and the owner can be a GitHub org.
PAT is associated with the user that created the PAT thus if the user leaves the company, the PAT will have no longer access to the org.

ArgoCD bot Communication with Github while posting diffs on PR

I am trying to configure ArgoCD bot to comment the diffs on PR. I have come so far to see that Argo bot is deployed in my system and pod/deployment being made. I enabled the debug flag also to see the logs in pod. The github app is installed in my repo and I have populated the .env file as well, so the pod is made without any issues.
I have loaded the secrets using the script that is provided in the repo and as instructed in the README. The .pem key is placed at my root folder.

This is my .env file (has dummy values, not real ones)

PORT=80
LOG_LEVEL=debug
KUBECTL_EXTERNAL_DIFF=
APP_ID=700908
[email protected]:user/app.git
GITHUB_TOKEN=dc78adfh28374hjshf8b061dcd1f69bc1389f4f6
WEBHOOK_SECRET=webhook-secret
PRIVATE_KEY_PATH=/home/private/key/argocd-bot/argocd-diff.private-key.pem
ARGOCD_SERVER=https://ARGO-IP/
ARGOCD_AUTH_TOKEN=adfadfadf34342sfsfsfs23aI

The following are the logs from the pod:

  params: {
    "per_page": 100,
    "baseUrl": "https://api.github.com",
    "request": {
      "timeout": 0
    }
  }
06:01:09.276Z DEBUG github: GitHub request: POST /app/installations/:installation_id/access_tokens - 201 Created (installation=198181122)
  params: {
    "installation_id": 198181122,
    "baseUrl": "https://api.github.com",
    "request": {
      "timeout": 0
    }
  }
06:01:10.314Z DEBUG github: GitHub request: GET /installation/repositories - 200 OK (installation=198181122)
  params: {
    "per_page": 100,
    "baseUrl": "https://api.github.com",

Also ARGOCD_AUTH_TOKEN it is recommended to generate an automation token using the /api/v1/projects/{project}/roles/{role}/token API but this is not found on my Argo Deployment. I am unable to find this endpoint.

When the PR is made in the repo, the events are received in the logs of the pod but the bot is unable to post the diff on the PR.
I also changed the internal ingress to the external one to see if there was any communication issue.
The slack channel doesnot have any pointers in this regard as well. I have tried it on multiple ArgoCD installations local and on cloud premises but circling back at the same issue again and again

collapse multiple spaces in comment into one

handle weird comments like this:
example comment:

argo     diff  -d  dir .    

Should work with pull requests from different user

Currently this only works for PRs from within the same repository.

This can easily be fixed by using the special github git references for pull requests e.g. refs/pull/42/head

Add SECURITY.md

The Argo maintainers recently agreed to require all Argoproj Labs project repositories to contain a SECURITY.md file which documents:

• Contact information for reporting security vulnerabilities
• Some minimal information about policies, practices, with possibly links to further documentation with more details

This will help direct vulnerability reporting to the right parties which can fix the issue.

You are free to use the following as examples/templates:

• Argoproj
• Workflows
• CD

Also, please note that in the future we are exploring a requirement that argoproj-labs projects perform a CII self-assessment to better inform its users about which security best practices are being followed.

Clean up new lines from bot comment

kubectl logs argocd-bot-6cbc5bbc84-9g5np
Received diff, processing; command: [ 'argo', 'diff', '-d', 'projects/go-test\r\n\r\n' ]

Project status

Is this project still being actively contributed to? It appears that commits have not been made for a long period of time. Is there an alternate approach that is suggested now over this?

GHE_HOST left empty as the doc said bot should use github.com but it does not do that

If we left GHE_HOST empty we hitting the issue as below

rgocd-bot-597b9df857-slcpw argocd-bot > probot run ./lib/index.js
argocd-bot-597b9df857-slcpw argocd-bot [before-after-hook]: "Hook()" repurposing warning, use "Hook.Collection()". Read more: https://git.io/upgrade-before-after-hook-to-1.4
argocd-bot-597b9df857-slcpw argocd-bot 07:25:21.079Z  INFO probot: Listening on http://localhost:8080
argocd-bot-597b9df857-slcpw argocd-bot 07:25:21.097Z DEBUG github: GitHub request: GET /app/installations - 500 Internal Server Error (installation=undefined)
argocd-bot-597b9df857-slcpw argocd-bot   params: {
argocd-bot-597b9df857-slcpw argocd-bot     "per_page": 100,
argocd-bot-597b9df857-slcpw argocd-bot     "baseUrl": "",
argocd-bot-597b9df857-slcpw argocd-bot     "request": {
argocd-bot-597b9df857-slcpw argocd-bot       "timeout": 0
argocd-bot-597b9df857-slcpw argocd-bot     }
argocd-bot-597b9df857-slcpw argocd-bot   }
argocd-bot-597b9df857-slcpw argocd-bot 07:25:21.098Z ERROR probot: Only absolute URLs are supported
argocd-bot-597b9df857-slcpw argocd-bot   HttpError: Only absolute URLs are supported
argocd-bot-597b9df857-slcpw argocd-bot       at fetch.then.then.catch.error (/home/argocd/argocd-bot/node_modules/@octokit/rest/lib/request/request.js:105:13)

Switch to using .ts instead of .js

• rename all files from .ts to .js
• switch to using ts-node

Docker build fails on npm run build

The docker image in Dockerhub was built with an older version of the ArgoCD image, so it pulls the Helm2 binary instead of Helm3. To get around that, I attempted to build my own version of the image to get the latest ArgoCD image and binaries.

However, I get a build failure running the npm commands in a docker build:

Step 18/18 : RUN npm install && npm run build && npm run test
 ---> Running in 4629bb1e0cbd
npm WARN deprecated [email protected]: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated [email protected]: use String.prototype.padStart()
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: this library is no longer supported

> [email protected] install /home/argocd/argocd-bot/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js

make: Entering directory '/home/argocd/argocd-bot/node_modules/dtrace-provider/build'
  TOUCH Release/obj.target/DTraceProviderStub.stamp
make: Leaving directory '/home/argocd/argocd-bot/node_modules/dtrace-provider/build'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN [email protected] No license field.

added 672 packages from 869 contributors and audited 673 packages in 23.718s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

> [email protected] build /home/argocd/argocd-bot
> tslint src/**/*.ts{,x} && tsc && cp -r ./src/sh ./lib/

node_modules/probot/lib/application.d.ts(34,20): error TS2314: Generic type 'WebhookEvent<T>' requires 1 type argument(s).
node_modules/probot/lib/context.d.ts(1,24): error TS2724: Module '"../../@octokit/webhooks"' has no exported member 'WebhookPayloadWithRepository'. Did you mean 'WebhookPayloadRepositoryImport'?
node_modules/probot/lib/context.d.ts(20,41): error TS2314: Generic type 'WebhookEvent<T>' requires 1 type argument(s).
node_modules/probot/lib/context.d.ts(29,24): error TS2314: Generic type 'WebhookEvent<T>' requires 1 type argument(s).
node_modules/probot/lib/github/index.d.ts(1,23): error TS2688: Cannot find type definition file for 'bunyan'.
node_modules/probot/lib/github/index.d.ts(2,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@octokit/rest/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(1,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@octokit/webhooks/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(16,20): error TS2314: Generic type 'WebhookEvent<T>' requires 1 type argument(s).
node_modules/probot/lib/wrap-logger.d.ts(1,23): error TS2688: Cannot find type definition file for 'bunyan'.
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] build: `tslint src/**/*.ts{,x} && tsc && cp -r ./src/sh ./lib/`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/argocd/.npm/_logs/2020-08-26T20_31_08_119Z-debug.log
The command '/bin/sh -c npm install && npm run build && npm run test' returned a non-zero code: 2

In case this was an out-of-date package issue, I tried updating npm packages via the instructions here, and while that did help with deprecation warnings, it still failed for the same reason, errors related to probot.

docker build output

Step 18/18 : RUN npm install && npm run build && npm run test
 ---> Running in ae90af191043

> [email protected] install /home/argocd/argocd-bot/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js

make: Entering directory '/home/argocd/argocd-bot/node_modules/dtrace-provider/build'
  TOUCH Release/obj.target/DTraceProviderStub.stamp
make: Leaving directory '/home/argocd/argocd-bot/node_modules/dtrace-provider/build'
npm WARN [email protected] No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 1018 packages from 912 contributors and audited 1019 packages in 12.849s
found 0 vulnerabilities


> [email protected] build /home/argocd/argocd-bot
> tslint src/**/*.ts{,x} && tsc && cp -r ./src/sh ./lib/

node_modules/probot/lib/application.d.ts(3,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@octokit/webhooks/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/application.d.ts(4,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@types/express/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/context.d.ts(1,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@octokit/webhooks/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/context.d.ts(2,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/deepmerge/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/github/logging.d.ts(1,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@types/bunyan/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(4,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@octokit/webhooks/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(5,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@types/bunyan/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(6,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@types/express/index"' can only be default-imported using the 'esModuleInterop' flag
node_modules/probot/lib/index.d.ts(7,8): error TS1259: Module '"/home/argocd/argocd-bot/node_modules/@types/ioredis/index"' can only be default-imported using the 'esModuleInterop' flag
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] build: `tslint src/**/*.ts{,x} && tsc && cp -r ./src/sh ./lib/`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/argocd/.npm/_logs/2020-08-26T20_34_27_005Z-debug.log
The command '/bin/sh -c npm install && npm run build && npm run test' returned a non-zero code: 2

Support Bitbucket PRs

We use bitbucket so I'd like for this to support BB PRs.

view_app_info.sh failed due to transport is closing

We tried to get the bot up and running. Even our webhook and subscribe events is correct but the bot didn't do anything. The argocd-bot pod has no logs. And if we mimic script ran it by hand it error out as below:

argocd@argocd-bot-6776676ffb-5n24f:~/argocd-bot$ argocd app list --plaintext
FATA[0000] rpc error: code = Unavailable desc = transport is closing
argocd@argocd-bot-6776676ffb-5n24f:~/argocd-bot$ argocd app get argo-events --plaintext
FATA[0000] rpc error: code = Unavailable desc = transport is closing 

argocd-bot to work using Azure Repos

The bot needs to be able to support Git repos in Azure DevOps