nytimes/drone-gae

Drone plugin for managing deployments and services on Google App Engine (GAE)

drone-gae

Drone plugin to manage deployments on Google App Engine.

Links

• Usage documentation
• Docker Hub release tags
• GitHub Actions Workflow runs
• Contributing documentation

Overview

This plugin is a simple wrapper around the appcfg.py and gcloud app commands, which makes it capable of making deployments in the standard environment or flexible environments with any language available.

The action configuration variable (shown below) can accept any action that you would normally call on appcfg.py or gcloud app. So far, it has been tested with update to deploy and set_default_version to migrate traffic in appcfg and gcloud app deploy for gcloud app, but it should also be capable of running helpful ops commands like update_indexes and update_cron.

To see a full list of configuration settings for the project, check out the GAE struct declaration.

To see the App Engine SDK and gcloud versions, check out the Dockerfile dependency download.

Drone versions compatibility

This plugin supports Drone 1.0+.

For usage, see these docs and main.go.

drone-gae's Issues

No URLMap entries found in application configuration on new php72 runtime (standard)

php 7.2 is now available on App Engine standard: https://cloud.google.com/appengine/docs/standard/php7/config/appref#handlers_element

I wanted to implement it for our apps as it would reduce our costs compared to a flexible environment.

Here's my app.test.yaml:

# Default GCloud App Engine deployment settings
runtime: php72

service: service-test

instance_class: B1
manual_scaling:
  instances: 1

env_variables:
  ENV: 'my env vars'

# GCloud DB settings
beta_settings:
  cloud_sql_instances: 'project:localization:service'

When using gcloud app deploy app.test.yaml --project=project my service works well.

But when using the drone plugin it crashes saying:

ERROR: (gcloud.app.deploy) An error occurred while parsing file: [/drone/src/bitbucket.org/mylocalname/service/app.yaml]
No URLMap entries found in application configuration
  in "/drone/src/bitbucket.org/mylocalname/service/app.yaml"

Here's the part responsible of the deploy in my .drone.yml:

deploy-test:
    image: nytimes/drone-gae
    action: deploy
    project: project
    app_file: app.test.yaml
    version: ${DRONE_BUILD_NUMBER}
    secrets:
      - source: GAE_CREDENTIALS_TEST
        target: GAE_CREDENTIALS
    when:
      event: deployment
      environment: test

First thing I found weird was that it seems to point to app.yaml in the error message although my file name is app.test.yaml. Renaming it didn't solve the issue and after checking it was working with a php runtime and flexible environment (pointing to app.yaml but the file was also named differently) so I believe this is not the issue.

I'm a bit lost and my knowledge is limited in Go.
Is it that it's new and not well interpreted by the plugin yet or am I missing something?

New SDK Out, Builds that passed in the morning are now failing

@jprobinson, I was build and deploying successfully this morning, but just started getting the below error message:

This build passed, and when i ran "restart" it failed.

It looks like there is a new SDK available for appengine that must have been released in the last few hours... digging in now to try and find out more information, but wanted to bring this issue to your attention.

Could not guess mimetype for frontend/build/asset-manifest.json.  Using application/octet-stream.
Could not guess mimetype for frontend/build/manifest.json.  Using application/octet-stream.
Could not guess mimetype for frontend/build/static/js/main.a5dca266.js.map.  Using application/octet-stream.
Could not guess mimetype for frontend/build/static/css/main.1d866fd8.css.map.  Using application/octet-stream.
Error 500: --- begin server output ---
<h3>Server Error</h3><p>A server error has occurred.</p>
There is a new release of the SDK available.

-----------
Latest SDK:
api_versions: ['1']
release: 1.9.66
supported_api_versions:
 go:
   api_versions: [go1, go1.6, go1.8]
 java7:
   api_versions: ['1.0']
 python:
   api_versions: ['1']
 python27:
   api_versions: ['1']
timestamp: 1516312066

-----------
Your SDK:
api_versions: ['1']
release: 1.9.63
supported_api_versions:
 go:
   api_versions: [go1, go1.6, go1.8]
 java7:
   api_versions: ['1.0']
 python:
   api_versions: ['1']
 python27:
   api_versions: ['1']
timestamp: 1509042494

-----------
Please visit https://developers.google.com/appengine/downloads
for the latest SDK
****************************************************************
--- end server output ---
error: exit status 1

[info] build failed (exit code 1)

Create tags of docker image

It would be great if we had tags for the docker image so folks could pin a specific version in there .drone.yml. This will avoid issues with changes to latest potentially breaking builds.

`gcloud app deploy` reads app.yaml config twice

In drone 0.7, under the deploy action attempts reads the app.yaml twice resulting in a failure

ERROR: (gcloud.app.deploy) [/drone/src/github.com/_/commit/0981727995bb55a6a2cab8b8fb8cd6c3a9655884/app.yaml] and [/drone/src/github.com/_/commit/0981727995bb55a6a2cab8b8fb8cd6c3a9655884/app.yaml] are both defining the service id [service-name]. All <service>.yaml files must have unique service ids.

As indicated by plugin output the command being run (in my case) is:

/google-cloud-sdk/bin/gcloud app deploy ./app.yaml --version v0-0-3 --project gcp-project --quiet --bucket=gs://service-name

running that command with copy & paste locally executes as expected, however there appears to be a zero-length space argument between deploy and ./app.yaml , as runninggcloud app deploy "" app.yaml will produce the error behavior locally.

The most likely point of failure seems to be https://github.com/NYTimes/drone-gae/blob/master/main.go#L272-L276

Feature: Add a `max_versions` configuration option to use in deployments

If supplied with a non-zero value along with the "deploy" action, after the deployment completes successfully, the plugin would delete any older versions beyond the "max version" count.

For example, if we added max_versions: 2 to deployment config and the deployed service had 4 total versions after the deployment completed, the plugin would remove the 2 older versions.

This is meant to deal with the limit of 210 versions for an App Engine project, which can be reached rather quickly if a large/active team is developing against the project. At the moment, reaching this 210 limit causes the plugin to error and developers must manually remove older versions from their projects. This should remove that manual intervention without requiring users to add any more moving parts to their deployment pipelines.

Failure to stop older versions of services

In several GAE flex deployments, passing the additional flag --stop-previous-version has failed to turn off/scale down previous versions to 0.

This can result in developers being charged for idle instances in the project.

This error does not happen every time, maybe around 1 in 5-10 times.

This may be related to:

• https://stackoverflow.com/questions/43430383/gcloud-app-deploy-does-not-remove-previous-versions
• GoogleCloudPlatform/app-gradle-plugin#107

Perhaps the SDK should be updated?

Example

Stops:

Does not stop:

Result:

Multi-line secrets are not redacted when "Running Command:"

Drone GAE Plugin built from 1e5e603cf8fe827a2ee0f1aced22361f00d6a714
Running Command: /google-cloud-sdk/bin/gcloud auth activate-service-account --key-file /tmp/gcloud.json
Activated service account credentials for: [[email protected]]
Running Command: /go_appengine/appcfg.py  --oauth2_access_token  [redacted] 
-A yyy -V zzz -E API_KEY: [redacted] -E CREDENTIALS: [redacted] "type": "service_account", "project_id": "xyz", "private...

Java support

I don't see Java listed in the plugin documentation as one of the supported languages for deployment. If it isn't supported, we could use this ticket to track potential Java support for the plugin. If it's supported, we can simply use this ticket to track a documentation update.

gcloud auth print-access-token transient error

Running Command: /google-cloud-sdk/bin/gcloud auth activate-service-account --key-file /tmp/gcloud.json
Activated service account credentials for: [[email protected]]
error creating access token: exit status 1

[info] build failed (exit code 1)

a few retries didn't work, then it did

also note

NAME
    gcloud auth print-access-token - print an access token for the active
        account
...
NOTES
    This command is an internal implementation detail and may change or
    disappear without notice.

see if there's a more official way to get the token, and/or get more info out of the error to understand why it fails, maybe have the plugin retry a few times

MaxVersions can error when trying to delete a version with an operation in progress

Steps to reproduce:

1. Deploy a GAE backend service with MaxVersions set to 1
2. Deploy the service again (maybe several times; the bug does not happen consistently). You will see an error from GCP that looks like this:

ERROR: (gcloud.app.versions.delete) Issue deleting version: [service-name/version-num]
--
[service-name/version-num]: ABORTED: Cannot operate on 
apps/project-name/services/service-name/versions/version-num 
because an operation is already in progress for 
apps/project-name/services/service-name/versions/version-num by {hash}.
error: exit status 1

After reaching out to GCP support to ask about this error, we received the following note from them:

If you are running the "remove versions" tool immediately after the version is deployed, we may still be performing operations internally (e.g. stopping the old version, performing traffic splitting for the new version, etc.). In cases where operations are still being performed on the old version, we will reject any other operations (in this case, your delete request). Whether other operations are occurring may also depend on how exactly you are using your CI/CD tool. I would recommend you to delete an old version more than 10 minutes after the new version is deployed and let me know if you still face the issue. We can do further digging in such a case.

Rather than waiting 10 minutes (which is a loong time lag to introduce to our Drone pipelines), this could be fixed by checking if an operation is already in progress and if so, waiting (or skipping) before starting a new operation on a version.

Add support for templating app.yaml files

Currently, there is no way of injecting secrets into the environment of gcloud app deploy commands via the CLI.

We should add a 'vars' and an app_template to the config that allows folks to declare a pass key/value pairs into a templateed app.yaml file.

MaxVersions tries to delete versions from other services when used with default service

Steps to reproduce:

1. Deploy the default service (we'll call it frontend) to the GAE project
2. Deploy the api service to the GAE project
3. Deploy a 2nd version of the api service to the GAE project
4. Deploy a 2nd version of the frontend service to the GAE project with maxVersions: 1

When you complete step 4 you will see it delete 2 services instead of 1. One from frontend and one from backend.

If you do the same but switch the deploy order of frontend (default) and backend (such that it's BFFB) then you will see the last step will only delete 1 service.

here is a sample drone where it will delete from other services yaml:

---
kind: pipeline
name: default

platform:
  os: linux
  arch: amd64

image_pull_secrets:
  - .dockerconfigjson

steps:

  - name: deploy_dev_frontend_default1
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: frontend.yaml #THIS IS THE frontend (default) SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

  - name: deploy_dev_backend1
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: backend.yaml #THIS IS THE API SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]


  - name: deploy_dev_backend2
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: backend.yaml #THIS IS THE API SERVICE
      dir: frontend_src
      max_versions: 2
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

  - name: deploy_dev_frontend_default2
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: frontend.yaml #THIS IS THE frontend (default) SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

Here's a sample of the default an non-default swapped with no issue:

---
kind: pipeline
name: default

platform:
  os: linux
  arch: amd64

image_pull_secrets:
  - .dockerconfigjson

steps:

  - name: deploy_dev_backend1
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: backend.yaml #THIS IS THE API SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

  - name: deploy_dev_frontend_default1
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: frontend.yaml #THIS IS THE frontend (default) SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

  - name: deploy_dev_frontend_default2
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: frontend.yaml #THIS IS THE frontend (default) SERVICE
      dir: frontend_src
      max_versions: 2
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

  - name: deploy_dev_backend2
    pull: if-not-exists
    image: nytimes/drone-gae
    settings:
      action: deploy
      app_file: backend.yaml #THIS IS THE API SERVICE
      dir: frontend_src
      max_versions: 1
      project: nyt-project-dev
    environment:
      GAE_CREDENTIALS:
        from_secret: google_credentials_dev
    when:
      event: [push]

Edit:
I've since confirmed that setting "service: default" in the drone.yaml or in the app.yaml will stop this plugin from deleting versions of all other services. However, that should probably happen without user having to specify.

upgrade gcloud

Heya, it looks like I'm getting a drone build fail because gcloud isn't recognizing my app.yml with new subnetwork config as below.

env:flex
runtime: custom
...
network:
  name: my-space-dev-net
  subnetwork_name: space

I'm thinking the fix would be to upgrade the gcloud + app engine SDKs.

Maybe 139.0.0 for gcloud since that's what I have, but I'm not sure about the App Engine SDK version – possibly 1.9.48?

ENV GOOGLE_CLOUD_SDK_VERSION=135.0.0
ENV GOOGLE_APP_ENGINE_SDK_VERSION=1.9.46

https://github.com/NYTimes/drone-gae/blob/master/Dockerfile#L3-L4