gocd-contrib/gomatic

A Python API for configuring GoCD

Gomatic

This is a Python API for configuring ThoughtWorks GoCD.

What does it do?

If you wanted to configure a pipeline something like that shown in the GoCD documentation then you could run the following script:

#!/usr/bin/env python
from gomatic import *

configurator = GoCdConfigurator(HostRestClient("localhost:8153"))
pipeline = configurator \
    .ensure_pipeline_group("Group") \
    .ensure_replacement_of_pipeline("first_pipeline") \
    .set_git_url("http://git.url")
stage = pipeline.ensure_stage("a_stage")
job = stage.ensure_job("a_job")
job.add_task(ExecTask(['thing']))

configurator.save_updated_config()

How does it work?

Gomatic uses the same mechanism as editing the config XML through the GoCD web based UI. The GoCdConfigurator gets the current config XML from the GoCD server, re-writes it as a result of the methods called, then posts the re-written config XML back to the GoCD server when save_updated_config is called.

Gomatic doesn't use the RESTful Go API because that is (currently) far too limited for our needs.

Limitations

We wrote it for our purposes and find it very useful; however, the current version has limitations (e.g. only really supports "Custom Command" task type) and allows you to try to configure GoCD incorrectly (which GoCD will refuse to allow). We will continue to work on it and will address its current limitations.

We believe it works for the following versions (as indicated by integration_test.py):

• 16.7.0-3819
• 16.8.0-3929
• 16.9.0-4001
• 16.10.0-413
• 16.11.0-418
• 16.12.0-435
• 17.1.0-4511
• 17.2.0-4587
• 17.3.0-4704
• 17.4.0-4892
• 17.5.0-5095
• 17.6.0-5142
• 17.7.0-5147
• 17.8.0-5277
• 17.9.0-5368
• 17.10.0-5380
• 17.11.0-5520
• 17.12.0-5626
• 18.1.0-5937
• 18.2.0-6228
• 18.3.0-6540
• 18.4.0-6640
• 18.5.0-6679
• 18.6.0-9515
• 18.7.0-9515
• 18.8.0-7433
• 18.9.0-7478
• 18.10.0-7703
• 18.11.0-8024
• 18.12.0-8222
• 19.1.0-8469
• 19.2.0-8641
• 19.3.0-8959
• 19.4.0-9155
• 19.5.0-9272
• 19.6.0-9515

We don't support the below versions anymore, however we did at some point in time, so it might still work in newer versions:

• 13.1.1-16714
• 13.2.2-17585
• 13.3.1-18130
• 13.4.0-18334
• 13.4.1-18342
• 14.1.0-18882
• 14.2.0-377
• 14.3.0-1186
• 14.4.0-1356
• 15.1.0-1863
• 15.2.0-2248
• 16.1.0-2855
• 16.2.1-3027
• 16.3.0-3183 [unsupported from 0.6.8 onwards]
• 16.4.0-3223 [unsupported from 0.6.8 onwards]
• 16.5.0-3305 [unsupported from 0.6.8 onwards]
• 16.6.0-3590 [unsupported from 0.6.8 onwards]

Install

We've written it using Python 2.7 but have been working on supporting python 3.5 and use tox to ensure that all unit tests are passing for both 2.7 and 3.5. You can install gomatic it using "pip":

sudo pip install gomatic

which will install the gomatic package.

Usage

We won't document all of the options. Most of the behaviour is covered by unit tests, so look at them.

Dry run

You can see what effect Gomatic will have on the config XML by using configurator.save_updated_config(save_config_locally=True, dry_run=True). If you have kdiff3 installed, Gomatic will open it showing the diff (if there is a difference) between the config XML before and after the changes made by the GoCdConfigurator. If you don't have kdiff3 installed, use a diff tool of your choice to diff the files config-before.xml vs config-after.xml.

Reverse engineering of existing pipeline

If you have already set up a pipeline through the UI and now want to retrospectively write a script to do the equivalent, you can get Gomatic to show you the script to create an existing pipeline:

python -m gomatic.go_cd_configurator -s <GoCD server hostname> -p <pipeline name>

This mechanism can also be useful if you can't work out how to script something; you just make the change you want through the GoCD web based UI and then reverse engineer to see how to do it using Gomatic. Bear in mind that Gomatic does not currently support every configuration option available in GoCD, so it might not be possible to do everything you want to do.

Gotchas

• Gomatic does not prevent you from creating config XML that GoCD will not accept. For example, if you create a stage that has no jobs, Gomatic won't complain until you try to run save_updated_config, at which time the GoCD server will reject the config XML.
• Gomatic does not check that the version of GoCD it is configuring supports all the features used. For example, versions of GoCD before 15.2 do not support encrypted environment variables on stages and jobs.

Developing Gomatic

You need to install Python's virtualenv tool and create a virtual environment (once):

pip install virtualenv

Then, to create the virtual environment, either:

• install autoenv and cd into the root directory of Gomatic

Or:

• execute .env in the root directory of Gomatic and then execute source venv/bin/activate

Then, if you are using IntelliJ IDEA:

1. File -> Project Structure -> Project SDK
2. New -> Python SDK -> Add local
3. select .../gomatic/venv/bin/python

Run the tests

Unit tests:

1. pip install -r requirements.txt
2. ./build.sh

Integration tests (takes a long time to download many versions of GoCD) (requires docker to be installed in order to run):

1. python -m unittest tests.integration_test

Contributing to Gomatic via pull request

To have the best chance of your pull request being merged, please:

1. Include tests for any new functionality
2. Separate out different changes into different pull requests
3. Don't delete any existing tests unless absolutely necessary
4. Don't change too much in one pull request

CI and releasing packages to pypi

Gomatic uses travis-ci to run the unit tests and deploy to pypi. Only tagged commits are deployed to pypi.

1. update value of version in setup.py
2. git tag -a v<version> -m 'version <version>' (e.g. git tag -a v0.3.10 -m 'version 0.3.10')
3. git push origin --tags

Alternatively, run:

1. bumpversion [major|minor|patch]
2. git push origin --tags

gomatic's Issues

Elastic agents are broken in gocd 18.1.0

With gocd 18.1.0 released, one of the changes made was to move the elastic profiles out of the server config. This breaking the elastic profiles being created with gomatic against gocd 18.1. The error I get trying to update my elastic profile:

File "/usr/lib/python3.6/site-packages/gomatic/go_cd_configurator.py", line 296, in save_updated_config
    self.__host_rest_client.post('/go/admin/restful/configuration/file/POST/xml', data, headers)
  File "/usr/lib/python3.6/site-packages/gomatic/go_cd_configurator.py", line 335, in post
    raise RuntimeError("Could not post config to Go server (%s) [status code=%s]:\n%s" % (url, result.status_code, message))
RuntimeError: Could not post config to Go server (http://localhost:8153/go/admin/restful/configuration/file/POST/xml) [status code=409]:
Invalid content was found starting with element 'elastic'. One of '{mailhost}' is expected.

I could move the elastic config to be out of the <server> tag but it would break for all the older gocd versions.

upcoming lock behaviour implementation

See gocd/gocd#3943

What is the state of this project?

Recent discussion on #10 has lead to my, and others, wondering the state of this project. Is this project still maintained? If so, who are the maintainers?

If this package is not maintained, can the following steps be taken?

1. Now: Update the README to reflect the state of the project, and request new maintainers.
2. Later: Transfer ownership of the project (including the PyPI package) to new maintainers.
3. Later: Update the README of this repo to point to the location of the new repo.

Define constants with the possible values for "lockBehaviour"

The three possible values are:

- unlockWhenFinished
- lockOnFailure
- none

As described in https://docs.gocd.org/current/configuration/admin_lock_pipelines.html

Should we support reverse engineering?

Just want to hear the ideas out there;

There is a lot of effort that went into supporting reverse engineering of pipelines and to keep supporting it we need to be a lot more thorough reviewing PRs and spend time testing and making sure that all new things are reversible. At the same time, we have a lot to catch up: config repos, auth plugins, roles.

Is reverse engineering a feature we want to keep in gomatic? Is the effort on maintaining it compatible with the value it brings?

Flaky tests with py35

The py35 tests are flaky in travis but I can't get them to fail in my box. If you have the time to try them out in your box and find the root cause that would be appreciated!

Config repo tag: Attributes changed

In GoCD 17.9 (coming next week), the <config-repo/> tag has changed a bit. The changes are explained gocd/gocd#3648 and gocd/gocd#3671.

I'm not sure how you handle different versions, but for 17.9 at least, something around this will probably have to change.

From what I can see in the GoCD code, the ID gets generated automatically, if not provided.

/cc @dudadornelles

Environment section in moved in config

When I try the example in the README at https://github.com/SpringerSBM/gomatic or the output from python -m gomatic.go_cd_configurator -s ... -p ... with dry_run=False, I get the following error with Go Version: 15.2.0:

  File "/usr/local/lib/python2.7/dist-packages/gomatic/go_cd_configurator.py", line 1102, in post
    raise RuntimeError("Could not post config to Go server (%s):\n%s" % (url, message))
RuntimeError: Could not post config to Go server (http://xxxx/go/admin/restful/configuration/file/POST/xml):
Invalid content was found starting with element 'pipelines'. One of '{agents}' is expected.

The "before" file had the structure below:

<?xml version="1.0" ?>
<cruise ...>
    <server .../>
    <pipelines group="defaultGroup">
           ...
    </pipelines>
    <environments>
           ...
    </environments>
    <agents>
           ...
    </agents>
</cruise>

The "after" file had this structure:

<?xml version="1.0" ?>
<cruise ...>
    <server .../>
    <environments>
           ...
    </environments>
    <pipelines group="defaultGroup">
           ...
    </pipelines>
    <agents>
           ...
    </agents>
</cruise>

I.e. the environments section moved up and appear before the pipelines instead of below it.

If I understand, correctly, the configuration should validate against https://github.com/gocd/gocd/blob/master/config/config-server/resources/cruise-config.xsd which states the following sequence:

        <xsd:sequence>
            <xsd:element minOccurs="0" maxOccurs="1" name="server">
               ...
            <xsd:element minOccurs="0" maxOccurs="1" name="repositories" type="repositoriesType">
               ...
            <xsd:element minOccurs="0" maxOccurs="1" name="scms" type="scmsType">
               ...
            <xsd:element minOccurs="0" maxOccurs="1" name="config-repos" type="configReposType">
               ...
            <xsd:element minOccurs="0" maxOccurs="unbounded" name="pipelines" type="pipelinesType"/>
            <xsd:element minOccurs="0" maxOccurs="1" name="templates" type="templatesType">
               ...
            <xsd:element minOccurs="0" maxOccurs="1" name="environments" type="environmentsType">
               ...
            <xsd:element minOccurs="0" name="agents" type="agentsType">
               ...
        </xsd:sequence>

Broken ElasticAgent support (v4 vs v5 EA API)

Howdy,

Seems like the XML config format for ElasticAgents has changed with the release of 19.3.

What used to be ...

<elastic>
  <profiles>
    <profile  id="docker.unit-test" pluginId="cd.go.contrib.elastic-agent.docker">
      <property>
        <!--
          The plugin currently only supports the `Image` property,
          which allows you to select the docker image that the build should run with
        -->
        <key>Image</key>
        <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
      </property>
    </profile>
  </profiles>
</elastic>

is now

  <elastic>
    <agentProfiles>
      <agentProfile id="some-id" clusterProfileId="some-cluster-profile">
        <property>
          <key>PodSpecType</key>
          <value>yaml</value>
        </property>
        <property>
          <key>Privileged</key>
          <value>true</value>
        </property>
        <property>
          <key>Image</key>
          <value>gocd/gocd-agent-docker-dind:v19.7.0</value>
        </property>
        <property>
          <key>PodConfiguration</key>
          <value>apiVersion: v1
kind: Pod
metadata:
  name: gocd-agent-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  serviceAccountName: default
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: gocd/gocd-agent-docker-dind:v19.7.0
      securityContext:
        privileged: true</value>
        </property>
      </agentProfile>
    </agentProfiles>
    <clusterProfiles>
      <clusterProfile id="some-cluster-profile pluginId="cd.go.contrib.elasticagent.kubernetes">
        <property>
          <key>go_server_url</key>
          <value>https://gocd-server:8154/go</value>
        </property>
        <property>
          <key>kubernetes_cluster_url</key>
          <value>https://10.96.0.1:443</value>
        </property>
        <property>
          <key>security_token</key>
          <encryptedValue>AES:SOMEREALLYLONGSTRINGYAY!!!!!1one</encryptedValue>
        </property>
        <property>
          <key>namespace</key>
          <value>ci</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
  </elastic>

If I understood correctly, the higher level stanza (AgentProfiles, ClusterProfiles), are dependent on the GoCD version (and elastic agent API version, eg, v5) ... the exact confg of agent profile are dependent on the version the agent plugin (eg, kubernetes) ... which gomatic doesn't specifically care about.

So, I think this is mostly a question of detecting the GoCD server version, and using the right format. Agent specifics remain a user issue.

support git material "Destination Directory"

Add missing gocd versions to integration tests

We are missing the following versions in the integration tests:

17.9
17.10
17.11
17.12
18.1

Black list on materials is not extracted when reverse engineering manual pipelines

python -m gomatic.go_cd_configurator -s -p
was used.

Please release version 0.6.8

To use new version of GoCD we need the latest version of gomatic, I can see you created 0.6.8 but haven't released it yet. Could you please make a release so it can be installed with pip.

Modify artifacts definition to include the type attribute

There has been a schema migration to use the same xml element to represent different types of artifacts. Here's the PR that introduced the change - gocd/gocd#4278

<artifact src="foo" dest="bar" />
<test src="baz" />

gets converted to

<artifact type="build" src="foo" dest="bar" />
<artifact type="test" src="baz" />

GoCD version detection does not work for templates

I've encountered an issue with creating a template from code, consider this bit of config:

#!/usr/bin/env python
from gomatic import *

configurator = GoCdConfigurator(HostRestClient("localhost:8153"))
template = configurator \
    .ensure_replacement_of_template("first_pipeline")

stage = template.ensure_stage("a_stage")
job = stage.ensure_job("a_job")
job.add_task(ExecTask(['thing']))

job.ensure_artifacts({
    gocd.artifacts.Artifact.get_build_artifact('a_file')
})

configurator.save_updated_config()

If you are using a version of GoCD that is 18.3 or later then this will fail with the following error:

Traceback (most recent call last):
  File "test.py", line 16, in <module>
    configurator.save_updated_config()
  File "/Users/david.reedfrancis/workspace/jsains/tickets/smss/188/smartshop-services-gocd/.venv_Darwin_x86_64_2.7.15/lib/python2.7/site-packages/gomatic/go_cd_configurator.py", line 307, in save_updated_config
    self.__host_rest_client.post('/go/admin/restful/configuration/file/POST/xml', data, headers)
  File "/Users/david.reedfrancis/workspace/jsains/tickets/smss/188/smartshop-services-gocd/.venv_Darwin_x86_64_2.7.15/lib/python2.7/site-packages/gomatic/go_cd_configurator.py", line 346, in post
    raise RuntimeError("Could not post config to Go server (%s) [status code=%s]:\n%s" % (url, result.status_code, message))
RuntimeError: Could not post config to Go server (http://localhost:8153/go/admin/restful/configuration/file/POST/xml) [status code=409]:
"Type" is required for Artifact

Digging into the code it seems that there is a function in pipeline.py that switches how the artifact XML is created based on the GoCD version, it seems that the detection will only work for pipelines that have a group as a parent - https://github.com/gocd-contrib/gomatic/blob/master/gomatic/gocd/pipelines.py#L105

I don't think this code will ever work for templates.

Update PyPI credentials and create a new release

The PyPI credentials in .travis.yml still use the previous maintainer's credentials. Update these to use a current maintainer's credentials. setup.py will also need to be updated.

Additionally, create a new release and publish to PyPI.

Does not support Github SCM material introduced by gocd-build-github-pull-requests

https://github.com/ashwanthkumar/gocd-build-github-pull-requests is used by many in combination with gocd to build and verify PRs

If you add any Github SCM material - the gomatic configuration will fail with:

  File ".../venv3/lib/python3.8/site-packages/gomatic/gocd/materials.py", line 28, in Materials
    raise RuntimeError("don't know of material matching " + ET.tostring(element, 'utf-8'))

For python3 you need to fix that to raise RuntimeError("don't know of material matching " + str(ET.tostring(element, 'utf-8'))) and you get a RuntimeError: don't know of material matching b'<scm ref="<uuid>" />\n '

I don't how the Github SCM works and if GitMaterial could be reused (the UI selectors in gocd look very similar)

Consider using the public API to pull the config XML?

Could you consider using the public API to get the latest config XML instead of the internal one?

gomatic/gomatic/go_cd_configurator.py

Lines 68 to 70 in 320aafd

	def __current_config_response(self):
	config_url = "/go/admin/restful/configuration/file/GET/xml"
	response = self.__host_rest_client.get(config_url)

Support for external artifacts

External artifacts differ from Build and Test artifacts in not having a src field. Instead an id and storeId field is present. In addition to that, a list of key-value properties is present, containing the plugin specific configuration.

An example XML configuration for docker-registry-plugin looks like this:

<artifact type="external" id="docker-image" storeId="docker-registry">
  <configuration>
    <property>
      <key>Image</key>
      <value />
    </property>   
    <property>
      <key>Tag</key>
      <value />
    </property>   
    <property>    
      <key>BuildFile</key>
      <value>buildfile.json</value>
    </property>   
  </configuration>
</artifact>

The fetch task for a docker-registry artifact looks like this:

<fetchartifact artifactOrigin="external" artifactId="docker-image" pipeline="build-docker-image" stage="build" job="build">
  <configuration>
    <property>
      <key>EnvironmentVariablePrefix</key>
      <value>DOCKERIZED_APP</value>
    </property>
    <property>
      <key>SkipImagePulling</key>
      <value>true</value>
    </property>
  </configuration>
  <runif status="passed" />
</fetchartifact>

The artifact store configuration looks like this:

 <artifactStores>
    <artifactStore id="docker-registry" pluginId="cd.go.artifact.docker.registry">
      <property>
        <key>RegistryType</key>
        <value>other</value>
      </property>
      <property>
        <key>RegistryURL</key>
        <value>private-registry:5000</value>
      </property>
      <property>
        <key>RegistryID</key>
        <value />
      </property>
      <property>
        <key>AWSAccessKeyId</key>
        <encryptedValue />
      </property>
      <property>
        <key>AWSSecretAccessKey</key>
        <encryptedValue />
      </property>
      <property>
        <key>AWSRegion</key>
        <value />
      </property>
      <property>
        <key>Username</key>
        <value>gomatic</value>
      </property>
      <property>
        <key>Password</key>
        <encryptedValue>secret</encryptedValue>
      </property>
    </artifactStore>
  </artifactStores>

Config Repo support

To support the pipeline-as-code plugin

https://docs.gocd.org/current/advanced_usage/pipelines_as_code.html

SVN not a supported material

It appears SVN is not a supported material. While trying to clone a template, it threw an exception as soon as it got to an SVN material in the XML.

Traceback (most recent call last):
  File "gocli.py", line 777, in <module>
    args.clonetemplate[0], args.clonetemplate[1])
  File "gocli.py", line 516, in clone_template
    self.configurator.save_updated_config()
  File "/opt/gocd/gomatic/go_cd_configurator.py", line 432, in save_updated_config
    config_after = prettify(self.config)
  File "/opt/gocd/gomatic/go_cd_configurator.py", line 74, in config
    self.reorder_elements_to_please_go()
  File "/opt/gocd/gomatic/go_cd_configurator.py", line 68, in reorder_elements_to_please_go
    pipeline.reorder_elements_to_please_go()
  File "/opt/gocd/gomatic/gocd/pipelines.py", line 530, in reorder_elements_to_please_go
    materials = self.materials
  File "/opt/gocd/gomatic/gocd/pipelines.py", line 397, in materials
    return [Materials(element) for element in elements]
  File "/opt/gocd/gomatic/gocd/materials.py", line 21, in Materials
    raise RuntimeError("don't know of material matching " + ET.tostring(element, 'utf-8'))
RuntimeError: don't know of material matching <svn dest="project_name" encryptedPassword="123456abc=" materialName="project_name" url="http://svn.example.com/svn/project_name/trunk" username="user" />

support timer "Run only on new material"

How far back should we consider backwards compatibility?

The integration tests at present test for all versions from 13.1 onwards. This currently means 30 distinct releases are being tested spanning 4 and a half years. This contributes to a very long integration test cycle that results in it not being run very often. Opening this issue to start discussion about if we want to limit the set of old versions we want to support.

Incompatibility with GoCD version 20.2.0

Everything worked fine on GoCD version 20.1.0 but since GoCD 20.2.0 release gomatic can't create pipelines as it seems the configuration file is changed and it's not compatible anymore. Are there any plans to support it?

Example response:
RuntimeError: Could not post config to Go server (https://DNS/go/api/admin/config.xml) [status code=400] (and result was not json): <Response [400]>

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Parse cruise.xml to identify invalid changes

--placeholder issue--

Attribute "dest" (destination directory) dropped from git materials.

When I have pipeline material like below in my config:

        <materials>
            <git dest="x" url="ssh://git..."/>
            <git dest="y" url="ssh://git..."/>
        </materials>

Gomatic drops the dest attribute, so that it becomes like this instead:

        <materials>
            <git url="ssh://git..."/>
            <git url="ssh://git..."/>
        </materials>