GoCD pipeline configuration as code
This is a GoCD server plugin which allows to keep pipelines and environments configuration in version control systems supported by GoCD (git, svn, mercurial, etc.). See this document to find out what GoCD's configuration repositories are.
- Setup
- File pattern
- Validation
- Format reference
- Format version
- Issues and questions
- Development
- License
Step 1: GoCD versions newer than 17.8.0 already have the plugin bundled. You don't need to install anything.
If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from
the releases page and place it on the GoCD server in
plugins/external directory.
Step 2: Follow the GoCD documentation to add a new configuration repository.
You can use the example repository at: https://github.com/tomzo/gocd-json-config-example.git as a reference.
In your config repo (tomzo/gocd-json-config-example.git in this case), ensure that your GoCD json config is suffixed with .gopipeline.json for pipelines and .goenvironment.json for environments. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.
Using this plugin you can store any number of pipeline or environment configurations in a versioned repository like git.
By default pipelines should be stored in *.gopipeline.json files
and environments should be stored in *.goenvironment.json files.
The file name pattern can be changed on plugin configuration page.
You can validate if proposed GoCD JSON changes will be accepted by the server. Currently, 2 options are available:
You may find this introductory blog post useful.
There is an ongoing effort to allow in-depth validation of configuration before pushing configuration to the source control. This is provided by GoCD's preflight API and gocd-cli.
You have several options to configure validation tools on your workstation:
- If you have a local docker daemon, use the gocd-cli-dojo image. Follow the setup instructions in the image readme.
- If you don't want to use docker, you'll need to setup
gocd-clion your host.
Either way you'll have gocd binary in your PATH or inside the docker container.
This will check general validity of the yaml file, without talking to the GoCD server:
gocd configrepo --json syntax ci.gopipeline.jsonThis command will parse and submit your json file to the configured GoCD server.
gocd configrepo preflight --json -r gocd-json-config-example *.gopipeline.json
Where -r is the configuration repository id, which you have earlier configured on GoCD server. You can check it on config repos page of your GoCD server, at /go/admin/config_repos. It is in the upper left corner of each config repo.
The pipeline configuration files should be stored in format similar to one exposed by GoCD API.
The format of environment configuration files is much simpler, you can find examples of correct environments below.
Please note that it is now recommended to declare the same format_version in each *.gopipeline.json or *.goenvironment.json file.
Supports format_version value of 9. In this version, support of ignore_for_scheduling for dependency materials has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.
Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 7 and 8. In version 7, support for properties has been removed. In version 8, support for mingle has been removed.
Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 6. In this version, support of allow_only_on_success for approval on stage has been added. Setting this attribute will ensure that the manual trigger will be allowed only if the previous stage is successful.
Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 5. In this version, support of username and encrypted_password for git and hg material has been added. In addition to that, hg will also support branch attribute.
Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 4. In this version, support has been added to control the display order of pipelines.
This server version also supports format_version of 3 and 2. Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 3. In this version fetch artifact format was changed to include artifact_origin.
This server version also supports format_version of 2. Using a newer format_version includes all the behavior of the previous versions too.
Supports format_version value of 2. In this version pipeline locking behavior was changed.
Configures a GoCD environment
{
"name": "dev",
"environment_variables": [
{
"name": "key1",
"value": "value1"
}
],
"agents": [
"123"
],
"pipelines": [
"mypipeline1"
]
}Environment variables is a JSON array that can be declared in Environments, Pipelines, Stages and Jobs.
Any variable must contain name and value or encrypted_value.
{
"environment_variables": [
{
"name": "key1",
"value": "value1"
},
{
"name": "keyd",
"encrypted_value": "v12@SDfwez"
}
]
}{
"format_version" : 1,
"group": "group1",
"name": "pipe2",
"label_template": "foo-1.0-${COUNT}",
"enable_pipeline_locking" : false,
"parameters": [
{
"name": "param",
"value": "parameter"
}
],
"tracking_tool": null,
"timer": {
"spec": "0 15 10 * * ? *"
},
"environment_variables": [],
"materials": [
...
],
"stages": [
...
]
}{
"format_version" : 1,
"group": "group1",
"name": "pipe-with-template",
"label_template": "foo-1.0-${COUNT}",
"enable_pipeline_locking" : false,
"template": "template1",
"parameters": [
{
"name": "param",
"value": "parameter"
}
],
"materials": [
...
]
}Please note:
- Pipeline declares a group to which it belongs
Expected since GoCD v17.12, you need to use lock_behavior rather than enable_pipeline_locking.
"lock_behavior" : "none"
lock_behavior can be one of:
lockOnFailure- same asenable_pipeline_locking: trueunlockWhenFinished-none- sameenable_pipeline_locking: false
When format_version is 4 (see above), the order of display of pipelines on the GoCD dashboard can be influenced by setting the display_order_weight property.
- This is an integer property and the pipelines in a pipeline group will be ordered by this value.
- The default value for this property is
-1. - Pipelines defined in GoCD's config XML will also default to -1.
- If multiple pipelines have the same
display_order_weightvalue, their order relative to each other will be indeterminate.
{
"name": "pipeline1",
"group": "pg1",
"display_order_weight": 10
},
{
"name": "pipeline2",
"group": "pg1",
"display_order_weight": -10
}In the above example, since both pipelines are in the same group, pipeline2 will be shown ahead of pipeline1. If any pipelines are defined in the GoCD config XML, then they will appear in between these two pipelines.
{
"spec": "0 0 22 ? * MON-FRI",
"only_on_changes": true
}{
"link": "http://your-trackingtool/yourproject/${ID}",
"regex": "evo-(\\d+)"
}DEPRECATION NOTICE: Since GoCD version 19.9 and format_version 8, this is no longer supported
{
"base_url": "https://mingle.example.com",
"project_identifier": "foobar_widgets",
"mql_grouping_conditions": "status > 'In Dev'"
}{
"name": "test",
"fetch_materials": true,
"never_cleanup_artifacts": false,
"clean_working_directory": false,
"approval" : null,
"environment_variables": [
{
"name": "TEST_NUM",
"value": "1"
}
],
"jobs": [
...
]
}{
"type": "manual",
"allow_only_on_success": true,
"users": [],
"roles": [
"manager"
]
}{
"name": "test",
"run_instance_count" : null,
"environment_variables": [],
"timeout": 180,
"elastic_profile_id": "docker-big-image-1",
"tabs": [
{
"name": "test",
"path": "results.xml"
}
],
"resources": [
"linux"
],
"artifacts": [
{
"source": "src",
"destination": "dest",
"type": "test"
},
{
"type": "external",
"id": "docker-release-candidate",
"store_id": "dockerhub",
"configuration": [
{
"key": "Image",
"value": "gocd/gocd-demo"
},
{
"key": "Tag",
"value": "${GO_PIPELINE_COUNTER}"
},
{
"key": "some_secure_property",
"encrypted_value": "!@ESsdD323#sdu"
}
]
}
],
"tasks": [
...
]
}There are 3 types of artifacts recognized by GoCD. Build and Test artifacts are stored on the GoCD server.
The source and the destination of the artifact that should be stored on the GoCD server must be specified.
{
"source": "src",
"destination": "dest",
"type": "build"
}{
"source": "src",
"destination": "dest",
"type": "test"
}Artifacts of type external are stored in an artifact store outside of GoCD.
The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available.
The external artifact store is referenced by the store_id. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration.
{
"type": "external",
"id": "docker-release-candidate",
"store_id": "dockerhub",
"configuration": [
{
"key": "Image",
"value": "gocd/gocd-demo"
},
{
"key": "Tag",
"value": "${GO_PIPELINE_COUNTER}"
},
{
"key": "some_secure_property",
"encrypted_value": "!@ESsdD323#sdu"
}
]
}DEPRECATION NOTICE: Since GoCD version 19.9 and format_version 7, properties are no longer supported
{
"name": "coverage.class",
"source": "target/emma/coverage.xml",
"xpath": "substring-before(//report/data/all/coverage[starts-with(@type,'class')]/@value, '%')"
}{
"name": "cobertura",
"path": "target/site/cobertura/index.html"
}Part of job object can be number of job to runs
"run_instance_count" : 6Or to run on all agents
"run_instance_count" : "all"Default is null which runs just one job.
All materials:
- must have
type-git,svn,hg,p4,tfs,dependency,package,plugin. - can have
nameand must havenamewhen there is more than one material in pipeline
SCM-related materials have destination field.
All scm materials can have filter object:
- for blacklisting:
"filter": {
"ignore": [
"externals",
"tools"
]
}- for whitelisting (since Go
>=16.7.0):
"filter": {
"whitelist": [
"moduleA"
]
}{
"url": "http://my.git.repository.com",
"branch": "feature12",
"filter": {
"ignore": [
"externals",
"tools"
]
},
"destination": "dir1",
"auto_update": false,
"name": "gitMaterial1",
"type": "git",
"shallow_clone": true,
"username": "user1",
"encrypted_password": "encrypted_value"
}For GoCD >= 19.4.0 and format_version: 5 and above:
You are advised to utilize username and encrypted_password for passing in material credentials as:
{
"url": "http://my.git.repository.com",
"branch": "feature12",
"username": "user1",
"encrypted_password": "encrypted_value"
}- Instead of
encrypted_passwordyou may specifypasswordbutencrypted_passwordmakes more sense considering that the value is stored in SCM. - Specifying credentials both in
attributesandurlwill result in a validation error e.g.INVALID MERGED CONFIGURATION Number of errors: 1+ 1. Ambiguous credentials, must be provided either in URL or as attributes.;; - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc
{
"url": "http://svn",
"username": "user1",
"encrypted_password": "encrypted_value",
"check_externals": true,
"filter": {
"ignore": [
"tools",
"lib"
]
},
"destination": "destDir1",
"auto_update": false,
"name": "svnMaterial1",
"type": "svn"
}Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.
{
"url": "repos/myhg",
"filter": {
"ignore": [
"externals",
"tools"
]
},
"destination": "dir1",
"auto_update": false,
"name": "hgMaterial1",
"type": "hg",
"username": "user1",
"encrypted_password": "encrypted_value",
"branch": "feature"
}For GoCD >= 19.4.0 and format_version: 5 and above:
You are advised to utilize username and encrypted_password for passing in material credentials as:
{
"url": "repos/myhg",
"username": "user1",
"encrypted_password": "encrypted_value"
}- Instead of
encrypted_passwordyou may specifypasswordbutencrypted_passwordmakes more sense considering that the value is stored in SCM. - Specifying credentials both in
attributesandurlwill result in a validation error e.g.INVALID MERGED CONFIGURATION Number of errors: 1+ 1. Ambiguous credentials, must be provided either in URL or as attributes.;; - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc
In addition to that, you can also leverage branch attribute to specify the branch for material
{
"branch": "feature"
}{
"port": "10.18.3.102:1666",
"username": "user1",
"encrypted_password": "encrypted_value",
"use_tickets": false,
"view": "//depot/dev/src... //anything/src/...",
"filter": {
"ignore": [
"lib",
"tools"
]
},
"destination": "dir1",
"auto_update": false,
"name": "p4materialName",
"type": "p4"
}Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.
{
"url": "url3",
"username": "user4",
"domain": "example.com",
"encrypted_password": "encrypted_value",
"project": "projectDir",
"filter": {
"ignore": [
"tools",
"externals"
]
},
"destination": "dir1",
"auto_update": false,
"name": "tfsMaterialName",
"type": "tfs"
}Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.
{
"pipeline": "pipeline2",
"stage": "build",
"name": "pipe2",
"type": "dependency",
"ignore_for_scheduling": false
}{
"package_id": "apt-repo-id",
"name": "myapt",
"type": "package"
}{
"scm_id": "someScmGitRepositoryId",
"destination": "destinationDir",
"filter": {
"ignore": [
"dir1",
"dir2"
]
},
"name": "myPluggableGit",
"type": "plugin"
}Since GoCD >= 19.2.0 defining new pluggable materials that are not defined
in the GoCD server is supported.
{
"plugin_configuration": {
"id": "plugin_id",
"version": "1"
},
"configuration": [
{
"key": "url",
"value": "[email protected]:tomzo/gocd-json-config-plugin.git"
}
],
"destination": "destinationDir",
"filter": {
"ignore": [
"dir1",
"dir2"
]
},
"name": "myPluggableGit",
"type": "plugin"
}This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:
...
"materials": [
{
"url": "https://github.com/tomzo/gocd-json-config-example.git",
"branch" : "ci",
"type": "git",
"name" : "mygit"
}
],
...And in server XML:
<config-repos>
<config-repo pluginId="json.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
</config-repo>
</config-repos>Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.
Using configrepo material type, above repetition can be avoided,
last example can be refactored into:
...
"materials": [
{
"type": "configrepo",
"name" : "mygit"
}
],
...Server interprets configrepo material in this way:
Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of
configrepomaterial.
Every task object must have type field. Which can be exec, ant, nant, rake, fetch, plugin
Optionally any task can have run_if and on_cancel.
run_ifis a string. Valid values arepassed,failed,anyon_cancelis a task object. Same rules apply as to tasks described on this page.
{
"type": "exec",
"run_if": "passed",
"on_cancel" : null,
"command": "make",
"arguments": [
"-j3",
"docs",
"install"
],
"working_directory": null
}{
"build_file": "mybuild.xml",
"target": "compile",
"type": "ant",
"run_if": "any",
"on_cancel" : null,
}{
"type": "nant",
"run_if": "passed",
"working_directory": "script/build/123",
"build_file": null,
"target": null,
"nant_path": null
}{
"type": "rake",
"run_if": "passed",
"working_directory": "sample-project",
"build_file": null,
"target": null
}{
"type": "fetch",
"artifact_origin": "gocd",
"run_if": "any",
"pipeline": "upstream",
"stage": "upstream_stage",
"job": "upstream_job",
"is_source_a_file": false,
"source": "result",
"destination": "test"
}{
"type": "fetch",
"artifact_origin": "external",
"run_if": "any",
"pipeline": "upstream",
"stage": "upstream_stage",
"job": "upstream_job",
"artifact_id": "upstream_external_artifactid",
"configuration": [
{
"key": "DestOnAgent",
"value": "foo"
},
{
"key": "some_secure_property",
"encrypted_value": "ssd#%fFS*!Esx"
}
]
}{
"type": "plugin",
"configuration": [
{
"key": "ConverterType",
"value": "jsunit"
},
{
"key": "password",
"encrypted_value": "ssd#%fFS*!Esx"
}
],
"run_if": "passed",
"plugin_configuration": {
"id": "xunit.converter.task.plugin",
"version": "1"
},
"on_cancel": null
}To build and test this plugin, you'll need java jdk >= 8.
If you have local java environment, then you may run all tests and create a ready to use jar with:
./gradlew test jarYou don't need to setup java on your host, if you are fine with using docker and Dojo. This is actually how our GoCD builds the plugin:
dojo "gradle test jar"
Assuming you already have a working docker, On OSX, you can install with homebrew:
brew install kudulab/homebrew-dojo-osx/dojo
A manual install is another option:
version="0.9.0"
# on Linux:
wget -O /tmp/dojo https://github.com/kudulab/dojo/releases/download/${version}/dojo_linux_amd64
# or on Darwin:
# wget -O /tmp/dojo https://github.com/kudulab/dojo/releases/download/${version}/dojo_darwin_amd64
chmod +x /tmp/dojo
mv /tmp/dojo /usr/bin/dojoThen enter a docker container with java and gradle pre-installed, by running following command at the root of the project:
dojo
-
If you have questions on usage, please ask them on the GoCD Google Groups forum or GitHub Discussions.
-
If you think there is a bug, or you have an idea for a feature and you are not sure if it's plugin's or GoCD fault/responsibity, please ask on the chat first too.
Please note this brief overview of what is done by the plugin:
- parsing files into json when GoCD server asks for it.
And this is done by the GoCD server:
- complex logic merging multiple config repo sources and XML
- validation of pipelines/stages/jobs/tasks domain
- any UI rendering
We use semantic versioning.
If you are submitting a new feature or patch, please bump the version in build.gradle.
Copyright 2019 Tomasz Sฤtkowski
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent