pantheon-systems/circleci-orb

Use CircleCI to push code to Pantheon Dev and Multidev Environments

circleci-orb's Issues

Add option for whether to switch to SFTP mode

updated issued description:

Switching to SFTP mode after pushing code adds time to the build process and may not be wanted. This would be best addressed by a change in the underlying Terminus Build Tools Plugin: pantheon-systems/terminus-build-tools-plugin#351

Figure out handling of sites without a live environment

terminus -n build:env:create "$TERMINUS_SITE.live" "$TERMINUS_ENV" --yes --clone-content assumes a live env exists. That is not true for all sites.

I see a few options for resolving this issue. There may be others.

1. Make source env an optional param on the job
2. Make the existence of the Live env required

Handle use case where GitHub holds only the contents of wp-content

We have a repo syncing to Pantheon that only contains the wp-content folder, and excludes all WordPress core files. So when this orb rsyncs to Pantheon here (https://github.com/pantheon-systems/circleci-orb/blob/master/src/%40orb.yml#L96), it deletes all of the WordPress core files from the multidev site. I've tried a few things with .gitignore but this doesn't seem ideal as it would interfere with upstreams. Is there a way around this?

Add Sass compliation example

Use https://github.com/stevector/wordpress-orb-demo to show compiling Sass in a custom job and then pushing with the Pantheon job.

Make multidev cleanup configurable

Hey folks! Another piece of feedback from my team on the Pantheon orb usage that I thought I'd share.

The build tools branch naming conventions and multidev management techniques (deleting stale multidevs to maintain space) are great sensible defaults, but they've seen some resistance among members of our team mostly for two reasons:

1. They prefer to name multidevs after the ticket number (which is our branch name)
2. They prefer to manually cleanup multidevs instead of having this automated (this is not a universal preference, but some see benefit in being able to access old multidevs after a project has shipped and there is a question about content/etc).

We would desire to be able to configure this behavior using orb parameters in a similar manner to the terminus clone environment, ability to not always clone content, etc.

This is another area where we could provide a pull request to implement if this feature were desirable, and it would again be backwards compatible by making the current behavior the default.

thanks!

Add more examples to readme

The readme should have a list of public repos that implement this orb.

Currently I'm using it in

• https://github.com/stevector/nerdologues-d8
• https://github.com/stevector/stevector-composer
• https://github.com/stevector/wordpress-orb-demo
• pantheon-systems/example-drops-8-composer#245

Run composer install

Does anyone have an example of configuring CircleCI to run composer install after this orb has finished?

I was relying on Pantheon to run composer install by adding build_step: true to pantheon.yml or pantheon.upstream.yml, but realized it conflicts with my Quicksilver script to import config after the sync_code workflow. I believe that Pantheon runs composer install after sync_code, which means I can't use it to trigger a config import because all my dependencies are missing at that point. There is somehow no composer_install workflow to use as a trigger as far as I can tell.

Compare/contrast with pantheon-systems/example-drops-8-composer

I'm using this orb and it's been pretty great. However, I'm wondering if there's something I'm missing out on by not using https://github.com/pantheon-systems/example-drops-8-composer instead. I appears my project initially was initially built on example-drops-8-composer because I have a .ci directory. Currently it serves no purpose as far as I can tell, but it got me thinking. Why are there two repos that seem to serve a similar purpose? What are the differences? Is there anything that the orb can do that example-drops-8-composer can't (or vice versa)? Is the difference mainly that the orb can be added to an existing project easily while example-drops-8-composer can't?

Thanks for any clarity.

[FEAT-1348] Option to not delete 'extraneous' files on rsync

We're working through some optimization steps to try to bring down our build times on CI builds, and one thing we're messing with is the concept of only building our theme when the theme files have changed, which could save us two minutes off many builds. Reviewing the orb code, I think the --delete in this line is what is blowing away our compiled theme assets when new builds happen. Could we look into making that behavior configurable while keeping the default? Perhap use an environment variable to determine whether or not to delete files, or a way to flag in additional exclude directories, etc would allow us to not disrupt current users but add in some flexibility here?

Push results in DB reset

I've set up the integration, which seems to work, but when I push new code, the db-update fails and my site redirects to the installer. Something is apparently wiping out the database in dev on each push. 😱

This shows up as a failed step in my build process:

#!/bin/bash -eo pipefail
terminus -n drush "$TERMINUS_SITE.$TERMINUS_ENV" -- updatedb -y
Warning: Permanently added '[appserver.dev.58920aaf-62bf-42b5-b885-70539047325c.drush.in]:2222,[35.193.88.141]:2222' (RSA) to the list of known hosts.
In BootstrapHook.php line 32:
  Bootstrap failed. Run your command with -vvv for more information.  
updatedb [--cache-clear [CACHE-CLEAR]] [--entity-updates] [--post-updates [POST-UPDATES]] [--no-cache-clear] [--no-post-updates] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--druplicon] [--xh-link XH-LINK] [--notify] [--] <command>
 [notice] Command: ************.dev -- drush updatedb [Exit: 1]
 [error]   
Exited with code exit status 1
CircleCI received exit code 1

The bootstrap of course fails because the DB has already been borked at this point, but I can't see where that'd be happening. After restoring the DB, the same db-update run from my local terminus works fine.

Could not determine authentication token for GitHub serivces. Please set GITHUB_TOKEN

I've followed the readme: https://github.com/pantheon-systems/circleci-orb

Yesterday pipeline builds were completing successfully. Today they are failing at the "Delete old Multidevs to make space for a potential new one" step with this error:

terminus -n build:env:delete:ci "$TERMINUS_SITE" --keep=2 --yes

 [notice] Rsync [e-mail]:code/build-metadata.json => /tmp/build-metadata.json
 
 [notice] Rsync [e-mail]:code/build-metadata.json => /tmp/build-metadata.json
 
 [error]  Could not determine authentication token for GitHub serivces. Please set GITHUB_TOKEN 
 
Exited with code exit status 1 
CircleCI received exit code 1 

The readme does not mention the GITHUB_TOKEN variable. I'm not sure how to interpret Github's documentation on it or where I would add this variable for a CircleCI integration. https://docs.github.com/en/actions/configuring-and-managing-workflows/authenticating-with-the-github_token

Github to Pantheon stuck with CircleCI

Hello all

For some reason my push to Pantheon using CircleCI got stuck at this stage:

 Initialized empty Git repository in /tmp/pantheon_repo/.git/
pantheon	ssh://[e-mail]:2222/~/repository.git (fetch)
pantheon	ssh://[e-mail]:2222/~/repository.git (push)
Warning: Permanently added '[codeserver.dev.86bd52c6-f874-4f97-b319-25ce381e8c7d.drush.in]:2222,[35.188.167.131]:2222' (RSA) to the list of known hosts.

Not sure why, it just dont pass this stage ..

I have added the terminus token via env variable on CircleCI UI.

And this is my orb:

version: 2.1
workflows:
  version: 2
  just_push:
      jobs:
      - pantheon/push
orbs:
  pantheon: pantheon-systems/[email protected]

I have also tried some others settings, i got the deploy process success on CircleCI but i dont see the commit on Pantheon, not sure what im missing, this is my last test:

# https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs
defaults: &defaults
  docker:
    - image: quay.io/pantheon-public/build-tools-ci:6.x
  working_directory: ~/example_drops_8_composer
  environment:
    #=========================================================================
    # In addition to the environment variables defined in this file, also
    # add the following variables in the Circle CI UI.
    #
    # See: https://circleci.com/docs/2.0/env-vars/
    #
    # TERMINUS_SITE:  Name of the Pantheon site to run tests on, e.g. my_site
    # TERMINUS_TOKEN: The Pantheon machine token
    # GITHUB_TOKEN:   The GitHub personal access token
    # GIT_EMAIL:      The email address to use when making commits
    #
    # TEST_SITE_NAME: The name of the test site to provide when installing.
    # ADMIN_PASSWORD: The admin password to use when installing.
    # ADMIN_EMAIL:    The email address to give the admin when installing.
    #
    # If your Pantheon environments use the security feature to lockdown
    # access with a HTTP Basic Authentication username and password, you may
    # also add any of the following environment variables to target specific
    # Pantheon environments. The value should be URL-encoded and follow the
    # format 'username:password' (without quotes).
    #
    # MULTIDEV_SITE_BASIC_AUTH
    # DEV_SITE_BASIC_AUTH
    # TEST_SITE_BASIC_AUTH
    # LIVE_SITE_BASIC_AUTH
    #
    # If all environments are locked using the same username and password you
    # may instead use the following environment variable in Circle CI UI.
    #
    # SITE_BASIC_AUTH
    #
    #=========================================================================
    TZ: "/usr/share/zoneinfo/America/Los_Angeles"
    TERMINUS_SITE:  ADDEDMYSITE
    TERMINUS_TOKEN: ADDEDMYTOKEN
    TERMINUS_ENV: dev
    # The variables below usually do not need to be modified.

version: 2
jobs:
  configure_env_vars:
    <<: *defaults

    steps:
      - checkout

      - run:
          # Set TERMINUS_ENV and related environment variables.
          # https://github.com/pantheon-systems/docker-build-tools-ci/blob/6.x/scripts/set-environment
          name: setup-environment-vars
          command: /build-tools-ci/scripts/set-environment

      - run:
          name: clone-bash-env
          command: cp $BASH_ENV bash_env.txt

      # https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs
      - persist_to_workspace:
          root: .
          paths:
            - bash_env.txt

  static_tests:
    <<: *defaults

    steps:
      - checkout

      - restore_cache:
          keys:
            - composer-cache-{{ checksum "composer.lock" }}
            - composer-cache-

  build_php:
    <<: *defaults
    steps:
      - checkout

      - restore_cache:
          keys:
            - composer-cache-{{ checksum "composer.lock" }}
            - composer-cache-
            - vendor-{{ checksum "composer.lock" }}
            - vendor-

      - run:
          # Set TERMINUS_ENV and related environment variables.
          # https://github.com/pantheon-systems/docker-build-tools-ci/blob/6.x/scripts/set-environment
          name: setup-environment-vars
          command: /build-tools-ci/scripts/set-environment

      - persist_to_workspace:
          root: .
          paths:
            - web
            - vendor
            - drush

  deploy_to_pantheon:
    <<: *defaults
    steps:
      - checkout

      # Attach the workspace
      - attach_workspace:
          at: /tmp

  composer_lock_updater:
    <<: *defaults
    steps:
      - checkout

      - run:
          # Set TERMINUS_ENV and related environment variables.
          # https://github.com/pantheon-systems/docker-build-tools-ci/blob/6.x/scripts/set-environment
          name: setup-environment-vars
          command: /build-tools-ci/scripts/set-environment

      - run:
          name: Authenticate with Terminus
          command: terminus -n auth:login --machine-token="$TERMINUS_TOKEN"

      - run:
          name: run composer lock updater
          command: set -e && terminus project:clu $TERMINUS_SITE

workflows:
  version: 2
  build_deploy_and_test:
    jobs:
      - configure_env_vars
      - static_tests
      - build_php
      - deploy_to_pantheon:
          requires:
            - static_tests
            - build_php

  scheduled_update_check:
    triggers:
       - schedule:
           cron: "0 0 * * *"
           filters:
             branches:
              only:
                - master
    jobs:
      - composer_lock_updater   
orbs:
  pantheon: pantheon-systems/[email protected]  

Consider options for reducing setup instructions.

The instructions in the readme for initial set up are more complex and error prone than I would like. terminus build:project:create was written when example-drops-8-composer had similarly complex set up instructions.

A new Build Tools Command could be made to support Orb set up. It could also be a stand alone CLI. Maybe a web service?

Make PR on Circle node orb for "npmbuild_and_persist"

Figure out how to support Quicksilver Pushback

Don't use an alternative branch in Terminus Build Tools

I'm make this issue here as a reminder for me to finish pantheon-systems/terminus-build-tools-plugin#202 and stop using that branch in this orb.

Consider release/tagging process recommended by CircleCI DevRel team

https://circleci.com/blog/creating-automated-build-test-and-deploy-workflows-for-orbs-part-2/

Add a more detailed explanation in Readme of git history and multidev details

The push job as currently written doesn't just push. It copies the state of the code in the GitHub repo and makes a new commit on top of the Pantheon repo. That should be explained in the readme.

The readme should also explain the implications of multidev availability.

Use a more meaningful commit message

The commits pushed to Pantheon all have the message "Build assets for $TERMINUS_ENV". Instead the message should probably incorporate the commit message used on github.

Pantheon Push Hangs on SSH key add

Hello pantheon folks,

When I run my build it hangs immediately following the initialization of the git repository in /tmp/pantheon_repo/.git. I get the warning about permanently adding the key to the list of known hosts and then it appears to prompt for the hosts password. After that it hangs, has anyone else run into this issue?

Add examples with other orbs

• Slack (The approval job for continuous delivery would be cool)
• Cypress
• NPM caching

Document env vars needed when retrofitting to existing non-build-tools sites

I've retrofitted this orb into a few sites run via Bitbucket + CircleCI + Pantheon that were using home-grown "Build tools inspired" CircleCI scripts that had become too cumbersome to maintain and slow in builds. It mostly worked flawlessly except that the Build Tools commands being used by the push command need to interact with the Bitbucket API to perform branch deletion tasks. These API calls were failing because I had not set the BITBUCKET_USER and BITBUCKET_PASS environment variables in CircleCI. Adding these env vars seems to have resolved the problem, but I had to go hunting in the code of the build tools plugin to discover what I was missing. I recommend adding these (and likely the GITHUB_TOKEN for users on GIthub, etc) to the readme to ensure retrofitters such as myself don't run into similar headaches.

Large Content --db-only

We have about 600MB database and 33GB files on a Drupal 9 site on Pantheon. The first run of creating a multidev, we get an error on the "Commit code to Pantheon and Push to Pantheon"

remote: 
remote: PANTHEON NOTICE:        
remote: 
remote: Skipping code sync, no Multidev environments were found for branch "pr-310".        
remote: 
 [notice] Creating multidev pr-310 for site ******************

The workaround is to re-run the workflow and it will pass. Only the first time run when PR is created we get this error.

I am assuming this is a timeout on cloning large content to the multidev in the terminus -n build:env:create "$TERMINUS_SITE.live" "$TERMINUS_ENV" --yes $CLONE_CONTENT --message="CI: $COMMIT_MSG" . This step takes about 10 mins to run and then timeout.

Is there a way to not clone files and only clone database (add --db-only as a parameter) to the multidev? I am thinking that we can use stage file proxy module for the files.

Update Pantheon Docs

Some docs should be updated to reference this orb

• https://pantheon.io/docs/guides/collaborative-development/
• https://pantheon.io/docs/guides/build-tools/

I'm sure there are more.

Feature request - What about hotfixes to live or to test envs?

Granted this is complex because there needs to be a way to know the intention via the github source. For an example from real life, the github repo can have three branches mapped respectively to each of the three pantheon envs, and a push to the test branch=push only to test environment. This could merge back down to the dev branch automatically; or make it optional so that the developer merges locally to dev branch and can take care of potential conflicts (then push back to dev).

Passing parameter to push job

I'm passing a false clone_content parameter however in circleci I'm getting this error. Not sure what I'm doing wrong

  compile_sass_and_push:
    jobs:
      - npmbuild_and_persist
      - pantheon/push:
          clone_content: false
          requires:
            - npmbuild_and_persist
          checkout: false
          pre-steps:
            - checkout
            - run: composer install --no-dev --optimize-autoloader

Document BitBucket

All of my testing has been on GitHub so far. I don't expect much difference in the end behavior. But the set up instructions need to be updated to account for BitBucket.

Add composer version 2

The docker image tag 6.x is using composer self-update --1. Is it possible to add a new release with tag 7.4.x that install composer 2?

multidev build failure

@stevector , thank you VERY much for this work, anything that makes learning CI/CD easier is hugely appreciated.

But, I hit a snag.

I created a branch, pushed to github, and made a pull request...

When we get to the Commit code to Pantheon repository and push to Pantheon part of the job, I get this error message

remote: PANTHEON NOTICE:        
remote: 
remote: Skipping code sync, no Multidev environments were found for branch "pr-1".        
remote: 
 [notice] Creating multidev pr-1 for site arsclan
 [notice] Created Multidev environment "pr-1"
 [notice] Enabled on-server development via SFTP for "pr-1"
 [error]  Could not figure out which git repository service to use with . 

I was watching the pantheon multidev dashboard, and I saw the PR-1 multidev being built, but then it vanished after this message, and the next 3 steps completed.

The job says it was successful, and github's pull request says everything is fine.

I do have a DEV, TEST and LIVE environment initialized, and any git pushes I was pushing right to the master branch worked perfectly.

The ONLY wrinkle that I'm adding to this, is that I'm using the pantheon drops 8 with composer with no CI documentation to set up the code base. https://pantheon.io/docs/guides/drupal-8-composer-no-ci/

But, again, pushing it to master worked, so, I hope thats not the problem.

Any thoughts anyone? Thank you again!

Old Multidevs not being deleted

During the build process there is a step called "Delete old Multidevs to make space for a new one". This does not work, which eventually causes builds to fail until I manually delete old Multidevs.

I can see the reason why but I'm not sure how to fix it. The build step output looks like this:

#!/bin/bash -eo pipefail
terminus -n build:env:delete:ci "$TERMINUS_SITE" --keep=2 --yes
 [notice] No environments matched the provided pattern "ci-".
CircleCI received exit code 0

Notice that it is looking for Multidevs that start with "ci-". None of the Multidevs created by this orb start with "ci-". They all start with "pr-". This makes some sense because our workflow is based on pull requests and looks like this:

• Branch off of master to work on a feature.
• Create a PR on Github to merge the feature branch back into master.
• CircleCI automatically creates a Multidev env representing the merge of the PR. The Multidev is named starting with "pr-".
• After QA on the Multidev, the PR is merged on Github. CircleCI automatically deploys to Dev env. (at the point the Multidev is no longer needed and could be deleted)

Is there any way for the "Delete old Multidevs to make space for a new one" step to delete Multidevs whose PR has been merged? If not, what is the purpose of this step?

Automated comments for environment creation

terminus build:env:create automates commenting on Pull Requests (really the commenting on the commit). See pantheon-systems/example-drops-8-composer#224 (comment) for an example comment.

For some reason this functionality is not working in my usage of this Orb. I imagine there's an environment variable or something different between example-drops-8-composer and example-wordpress-composer based projects and the ones I'm using.

I'm getting the error message [error] Could not figure out which git repository service to use with .. See https://circleci.com/gh/stevector/stevector-composer/1109 under Commit code to Pantheon repository and push to Pantheon for an example of this error.

Add option to change Master branch to Main

We would like to change our master branch on Github and would be great to have the option to set the default branch variable name. Thanks!

--clone-content is not optional

See https://github.com/pantheon-systems/circleci-orb/blob/master/src/%40orb.yml#L107

Users can not opt-out of --clone-content, which means the dev DB and files are copied to the multidev on each code push. This is an issue if there are existing changes to the DB on the multidev, as they are overwritten.

Add automatic integration tests

Pull requests to this repo result in new dev releases of the orb being published. To know if the new orb is safe, PRs should automatically be opened on other repos (example-drops-8-composer and example-wordpress-composer at least) that reference the newly published orb.

See stevector/nerdologues-d8#337 and stevector/nerdologues-d8#338 for examples of me manually introducing such a change by hash and by branch name.

Add checks for needed variables and authentication

Using this orb requires some environment variables to be set and ssh keys to be present. There should be helpful error messages as early as possible if any of them are not set.

Review orb with CircleCI DevRel team

This is a blocker for 1.0.0