baloise/gitopscli

GitOps CLI - a command line interface (CLI) to perform operations on git repositories with yaml files

GitOps CLI

GitOps CLI is a command line interface (CLI) to perform operations on GitOps managed infrastructure repositories, including updates in YAML files.

Quick Start

The official GitOps CLI Docker image comes with all dependencies pre-installed and ready-to-use. Pull it with:

docker pull baloise/gitopscli

Start the CLI and the print the help page with:

docker run --rm -it baloise/gitopscli --help

Features

• Update YAML values in config repository to e.g. deploy an application.
• Add pull request comments.
• Create and delete preview environments in the config repository for a pull request in an app repository.
• Update root config repository with all apps from child config repositories.

For detailed installation and usage instructions, visit https://baloise.github.io/gitopscli/.

Git Provider Support

Currently, we support BitBucket Server, GitHub and Gitlab.

Development

Setup

make init  # install dependencies, setup dev gitopscli, install pre-commit hooks, ...

Commands

make format  # format code
make format-check  # check formatting
make lint  # run linter
make mypy  # run type checks
make test  # run unit tests
make coverage  # run unit tests and create coverage report
make checks  # run all checks (format-check + lint + mypy + test)
make image  # build docker image
make docs  # serves web docs
make update  # update package dependencies

License

Apache-2.0

gitopscli's Issues

Provide additional author with parameter

Context

Although the commit created by gitopscli is usually done by CI with a technical user, the actual deployment is often triggered by a user (click in the CI gui or a commit in the application repository). It could be beneficial for the history to add this user as additional author to the deployment commit.

Solution Proposal

The deploy command would get two additional parameters for co-authors:

gitopscli deploy --git-provider github \
                          --username gitopscli \
                          --git-user CI-User \
                          --git-email [email protected] \
                          --co-author-name 'Mike Miller' \
                          --co-author-email [email protected]

The co-author values would be added to all commit messages created by gitopscli

Document (and automate?) release process

Write some documentation on how to release a new version.
Maybe use something like semantic release to automate the whole process in the first place?

Add option to create-preview for only updating a preview if it exists already

With an option --update-only the create-preview command could only update an existing preview (with a new tag) if it exists already.

This feature can be used to simply update a preview env on new pushes to the PR.

Alternatively a new update-preview command could be added.

Switch to dynamic linked library approach to avoid L-GPL copy left clauses

In my opinion we do not (yet) meet the necessary dynamic LGPL linking requirements to take advantage of the reduced requirements.

The Apache Legal docs also state that such a dependency is not possible.

As this is a standalone CLI application the site note for LGPL Combined Libraries is also not applicable.

IMHO we should either (preferably) switch to a dynamic linking approach (having an *.so or *.ddl or similar) respect LGPL-3.0 license and change projects license to (L)GPL-3.0 instead of Apache 2.0 and meet their requirements.

Create travis / circleci build

If we want to have the utility open source, we should use open source CI tools to build and publish to dockerhub

Helm release name max length exceeded

When using the default config the namespace, a namespace with 60 chars is beeing created.
This is fine for k8s, as 63 is the limit - however the helm chart release name must not exceed 53 characters:

Error: release name "gitops-nginx-demo-joachimprinzbach-patch-2-48f31385-preview" exceeds max length of 53

Gitlab merge labels and parameters

gitopscli creates merge requests in Gitlab without labels and parameters
https://github.com/baloise/gitopscli/blob/master/gitopscli/git_api/gitlab_git_repo_api_adapter.py#L57
I would like to add new parameter for gitopscli for Gitlab --gitlab-labels

labels in merge request

In documentation Gitlab it looks like
mr = project.mergerequests.create({'source_branch': 'cool_feature',
'target_branch': 'main',
'title': 'merge cool feature',
'labels': ['label1', 'label2']})

and also I can't configure merge parameters https://github.com/baloise/gitopscli/blob/master/gitopscli/git_api/gitlab_git_repo_api_adapter.py#L62

I would like to add parameter --gitlab-merge-parameters
It can be

• "merge_commit_message"
• "should_remove_source_branch"
• "merge_when_pipeline_succeeds"

for example:
mr.merge(merge_when_pipeline_succeeds=True)

If community is ok with this suggestion, I am ready to create the PR to add this functionality (however I can create and test it only for GItlab)

Links:
[python-gitlab]: https://python-gitlab.readthedocs.io/en/stable/gl_objects/merge_requests.html

Why YAML?

Why is the config specified in YAML format and not in more flexible plain Python? Is this for security reasons?

Create more tests

add some unit and integration tests

Create preview enhancements

• provide error handling and usefull log messages and return code when params are not correct
• branch not available in apps repo #72
• org / repo combination not available #49
• .gitops.config.yaml not available in apps repo #75
• .gitops.config.yaml not readable, property missing #75
• root org / root repo org from .gitops.config.yaml not available #49
• git_provider / git_provider_url not valid #49
• pr with pr_id not available #72
• comment with id parent_id not available
• any error while calling git server api #49, #66
• preview-template for app not available #75
• no route host available #75
• property to replace not available in values.yaml #48
• any error while performing git operations #66, #67
• replace logic should work for non helm based projects -> separate issue for this feature #74

Discussion: Future of GitOps CLI to transform to a developer and end-user centric CLI

Currently, the purpose of the GitOps CLI is to be used in a CI system to copy/change YAML and talk to the Git provider API. As already indicated in #104 I would like to discuss the future of the CLI to make the whole GitOps ecosystem around it more available to developer and new users.

Bootstrapping

A new user should be able to bootstrap the whole GitOps ecosystem using the CLI. This includes:

• Provisioning of the app-of-apps repository (apps-root-config)
• Provisioning the cluster-infra-apps repository
• Installation and configuration of Argo CD in the cluster-infra-apps repository
• Add CLI commands to add new config repositories
• Possible additions:
  • Installation support for SealedSecrets
  • Make the use of the GitOps ecosystem usable without the need to configure a CI system (apps-sync, preview envs) add support for a REST API which would be configured as webhook
  • Support for Monitoring Stack

Context Support

Currently a lot of the commands would be hardly usable if they’re used outside of a CI system due to the high amount of arguments. As I would like to see the GitOps CLI as a utility for developers to make common task easily executable. To reduce the amount of arguments which are required the context should be stored in a config file. As design inspiration I would orient on how kubectl or oc manage their contexts. This would include:

• Support for a login/init process which creates a config file storing:
  • References to the app-of-apps repository
  • Git credentials
  • User / Email – maybe taken from .gitconfig
  • Available config repositories to the user
• Support for context switching (between app-of-apps repository or config-repositories)

For the use in a CI system a config file could simply be passed either through a parameter or an env variable.

Developer Support

Besides the existing commands like create-preview the following features would round off the experience of the GitOps ecosystem

• Add reflection commands like get config-repos or get environments including the health status from Argo CD
• Add environment creation support like create environment –name demo-test –image org/app:latest –host app.example.tld. Some defaults like which environment template should be used could be stored in the app-of-apps repository (reference to generic-chart)
• Add environment deletion support
• Add promotion support (promote test to int) like promote –from demo-test –to demo-prod )
• Provide debugging utilities which could help identify issues of the deployment (duplicate environment name, Argo CD Health status etc.)
• Environment migration / duplication support (e.g. moving an environment between two config repositories, create new environments based on an existing one)

The order of how I wrote the ideas have nothing to do with the possible priority. What are you guy thoughts about how we should move on? @christiansiegel @joachimprinzbach

Ensure commits are signed off

I took the contributing guidelines from our "official" template: https://github.com/baloise/repository-template-java/blob/master/CONTRIBUTING.md

Those guidelines require all contributors to sign off their commit: https://baloise-incubator.github.io/gitopscli/contributing/#sign-your-work-developer-certificate-of-origin

We should automatically check that.

Allow token based authentication

GitHub will deprecate basic auth in the future:

Hi @christiansiegel,

You recently used a password to access an endpoint through the GitHub API using PyGithub/Python. We will deprecate basic authentication using password to this endpoint soon:

https://api.github.com/repositories/235198591

We recommend using a personal access token (PAT) with the appropriate scope to access this endpoint instead. Visit https://github.com/settings/tokens for more information.

Thanks,
The GitHub Team

Push additional docker tag with <major> and <major>.<minor> versions

A docker tag with only <major> and <major>.<minor> version should be published, so the consumers don't have to update their tags for every minor or patch release.

I created a PR in the semantic-release docker plugin: felixfbecker/semantic-release-docker#35

Leave incubator context and mature to top-level github-Orga

As this is actively used in production I'd like to propose to leave the incubator context and mature to our top-level github-Orga - what do you think?

State installation prerequesites in docs

In #139 @christiansiegel wrote:

Some things we might look into before releasing on PyPI:

• support and tests for more than one python version (currently only 3.8 is used in the docker build/test; I think even 3.7 doesn't work at the moment)
• support windows (currently the temporary directory for git checkout is hardcoded to /tmp)
• add proper error message if there is no git installed (currently probably results in an ugly exception)

Currently the Setup Page does not indicate any specific prerequisites for installing from source - this should be changed for usability.

Use recent version of code formatter

Upgrade black formatter to fix dependency issues. Details see #176

Create / Release GitOps CLI as a binary

The GitOps CLI should be released as a self contained binary so it could be used without Docker or Python installed locally.

It would be the best if it would be included in the release step.

To create a binary for a Python application I tested PyInstaller which worked great so far:

pip3 install pyinstaller
# Move to the gitopscli repository
pyinstaller gitopscli/__main__.py -n gitopscli --onefile

Afterwards, the executable binary is available under dist/gitopscli

Releasing the GitOps CLI as binary would open the door to add developer. I'm thinking about commands like:

gitopscli login --git-server bitbucket.example.tld --root-config <path-to-root-config>
gitopscli create deployment --name demo-test --image=org/myimage:latest --host=app.example.tld --config-repo example-non-prod ...
gitopscli promote --from demo-test --to demo-int

etc.

Error while parsing .gitops.config.yaml

Can be fixed by using Strings: baloise-incubator/gitops-nginx-demo@510d9ba

Error:

INFO clone: Cloning repository: https://github.com/baloise-incubator/gitops-nginx-demo.git
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/io_api/yaml_util.py", line 20, in yaml_file_load
    return YAML_INSTANCE.load(stream)
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/main.py", line 341, in load
    return constructor.get_single_data()
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 78, in get_single_node
    document = self.compose_document()
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 101, in compose_document
    node = self.compose_node(None, None)
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 138, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 218, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 138, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 211, in compose_mapping_node
    while not self.parser.check_event(MappingEndEvent):
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/parser.py", line 140, in check_event
    self.current_event = self.state()
  File "/opt/venv/lib/python3.8/site-packages/ruamel/yaml/parser.py", line 585, in parse_block_mapping_key
    raise ParserError(
ruamel.yaml.parser.ParserError: while parsing a block mapping
  in "/tmp/gitopscli/c1555293-f3d2-42e7-bcf1-ff6e9bfe0995/repo/.gitops.config.yaml", line 4, column 3
expected <block end>, but found '<scalar>'
  in "/tmp/gitopscli/c1555293-f3d2-42e7-bcf1-ff6e9bfe0995/repo/.gitops.config.yaml", line 4, column 28

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/bin/gitopscli", line 33, in <module>
    sys.exit(load_entry_point('gitopscli==4.8.0', 'console_scripts', 'gitopscli')())
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/__main__.py", line 14, in main
    command.execute()
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/commands/create_pr_preview.py", line 59, in execute
    create_preview_command.execute()
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/commands/create_preview.py", line 43, in execute
    gitops_config = self.__get_gitops_config()
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/commands/create_preview.py", line 87, in __get_gitops_config
    return load_gitops_config(self.__args, self.__args.organisation, self.__args.repository_name)
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/commands/common/gitops_config_loader.py", line 13, in load_gitops_config
    gitops_config_yaml = yaml_file_load(gitops_config_file_path)
  File "/opt/venv/lib/python3.8/site-packages/gitopscli/io_api/yaml_util.py", line 22, in yaml_file_load
    raise YAMLException(f"Error parsing YAML file: {file_path}") from ex
gitopscli.io_api.yaml_util.YAMLException: Error parsing YAML file: /tmp/gitopscli/c1555293-f3d2-42e7-bcf1-ff6e9bfe0995/repo/.gitops.config.yaml```

Switch to personal access tokens for Github

Github is deprecating password auth: https://developer.github.com/changes/2020-02-14-deprecating-password-auth/

Automate gh-pages build

Currently we manually have to tun make docs (= mkdocs gh-deploy) to build the website and commit it to branch gh-pages. We should automate this with travis on every push to master.

Can't change YAML values in an array

In the documentation is an example for changing a value which is part of an array (https://baloise.github.io/gitopscli/commands/deploy/)

--values "{frontend.tag: 1.1.0, backend.tag: 1.1.0, backend.env.[0].value: bar}"

For me, this results in the following error:

gitopscli deploy: error: argument --values: invalid YAML value: '{frontend.tag: 1.1.0, backend.tag: 1.1.0, backend.env.[0].value: bar}'

Looks like the underlying YAML library doesn't support the syntax

>>> from ruamel.yaml import YAML, YAMLError
>>> YAML().load("{frontend.tag: 1.1.0, backend.tag: 1.1.0, backend.env.[0].value: bar}")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 341, in load
    return constructor.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 78, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 101, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 138, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 211, in compose_mapping_node
    while not self.parser.check_event(MappingEndEvent):
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 140, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 726, in parse_flow_mapping_key
    raise ParserError(
ruamel.yaml.parser.ParserError: while parsing a flow mapping
  in "<unicode string>", line 1, column 1:
    {frontend.tag: 1.1.0, backend.ta ... 
    ^ (line: 1)
expected ',' or '}', but got '['
  in "<unicode string>", line 1, column 55:
     ... backend.tag: 1.1.0, backend.env.[0].value: bar}
                                         ^ (line: 1)

I saw in the code that there is some kind of handling for the array syntax (

gitopscli/gitopscli/io_api/yaml_util.py

Line 49 in 0c5dad9

is_array = _ARRAY_KEY_SEGMENT_PATTERN.match(current_key_segment)

)
but it is not possible to pass the YAML loading.

How is this intended to work? 😅

Release on PyPI

As this package already is installable via pip, semver-versioned >=1.0.0, open source licensed - why not release on PyPI?

Allow password input via stdin or env variable

...so that the password is not printed in the CI log.

Add PR Comment enhancements

• remove unneeded params (branch, create-pr, auto-merge) #42
• provide error handling and usefull log messages and return code when params are not correct
• username / password invalid #49
• org / repo combination not available #49
• git_provider / git_provider_url not valid #49
• pr with pr_id not available
  • Current message: ERROR:atlassian.rest_client:Received: 404
    Not Found
    Pull request 2 does not exist in APPS/docker-sample-nginx.
• comment with id parent_id not available
  • Current message: ERROR:atlassian.rest_client:Received: 404
    Not Found
    Comment 200 does not exist.
• any error while calling git server api
  • see current messages above

Do we really need baloise specific CI?

Why do we need baloise specific ci.groovy and Dockerfile.baloise?

[deploy]: not respecting special characters

The deploy command is not respecting the official YAML-Notation of special characters.

See: https://stackoverflow.com/a/22235064/5535773

We're not able to deploy (espacially upgrade --install) our configmap which includes values like:

• test://user:[email protected]:PORT or
• http://service

To be able to apply dynamic configmaps here's an example like helm v3 is suggesting:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app
  namespace: {{ .Release.Namespace }}
{{- with .Values.config.data }}
data:
  {{- toYaml . | nindent 2 }}
{{- end }}

If you try run the deploy command it overrides all double quoted config values from:

"test://user:[email protected]:PORT" -> test://user:[email protected]:PORT

gitopscli should check if yaml special characters are available and then decide to take the quoted or not quoted value.

Allow updating of yaml lists

• should we support adding an entry? And therefore also deleting?
• or just replacing the whole list with a new one?

Sync apps enhancements

• remove unneeded params (branch, create-pr, auto-merge) #42
• provide error handling and usefull log messages and return code when params are not correct
• username / password invalid #49
• org / repo combination not available #49
• root-org / root-repo combination not available #49
• git_provider / git_provider_url not valid #49
• any error while performing git operations
• bootstrap/values.yaml not found

Remove branch after merging PR

We should at least provide an option to delete the PR branch. Maybe it could be enabled by default.

create-preview: only 1 commit for initial create + update

create-preview currently creates 2 commits for a new preview, e.g.:

Create new preview environment for 'demo' and branch 'b037364/indexhtml-1590668567421'.
Update preview environment for 'demo' and branch 'b037364/indexhtml-1590668567421'.

The first one still contains the placeholders for image tag and route -> usually invalid values. IMO we should combine both steps into one commit.

Syncing multiple apps results in YAML anchors

If syncing multiple applications at the same time YAML anchors are used instead of new emtpy objects.

See: baloise-incubator/okd4-apps-root-config@618568e

Is is due to dict.fromkeys second argument value is always referencing the same object
https://github.com/baloise/gitopscli/blob/master/gitopscli/commands/sync_apps.py#L86

The code needs to be changed so that always a new object ({}) instead of the same one is used.

Relating stackoverflow question: https://stackoverflow.com/questions/15516413/dict-fromkeys-all-point-to-same-list

Refactor .gitops.config.yaml

Currently the .gitops.config.yaml file looks like this:

deploymentConfig:
  # The organisation name of your deployment repo
  org: deployments
  # The repostiory name of your deployment repo
  repository: deployment-config-repo
  # The name of the application (name of the folder in `.preview-templates`)
  applicationName: app-xy

previewConfig:
  route:
    host:
      # Your router host
      # {SHA256_8CHAR_BRANCH_HASH} gets replaced by a shortened hash of your preview_id
      template: app.xy-{SHA256_8CHAR_BRANCH_HASH}.example.tld
  replace:
    # Paths that should be replaced in the `values.yaml`
    - path: image.tag
      variable: GIT_COMMIT # this is the git hash of your app repo
    - path: route.host
      variable: ROUTE_HOST # this is the resolved host.template from above

Proposed changes:

• Add an apiVersion field so we can introduce breaking changes (at least required for the proposed changes)
• Rename deploymentConfig to configRepository or configRepo
• Drop previewConfig.route as it is OpenShift specific and not flexible
• Add templating support to previewConfig.replace[].variable to make the template feature which currently lives under previewConfig.route.host.template available in a more flexible manner. Maybe we should drop previewConfig.replace[].variable as well and name it previewConfig.replace[].value with "${VAR}" support .
• Rename the available SHA256_8CHAR_BRANCH_HASH variable to BRANCH_NAME_SHA_SHORT or similar - the current name is not rememberable.

Spec for Config

I did not find a spec for the Config in this repo's docs. There seems to be some validation in https://github.com/baloise/gitopscli/blob/master/gitopscli/gitops_config.py

With a JSON-Schema a human readable version could be generated (there are several generators) from the same source a validator could use. Non-JSON data can be validated this way, too.

Add tests for commands

Add unit tests for every command..
..so that bugs like #80 don't happen anymore 😬

Support previews for non Helm based projects

create-preview only allows replacements in values.yaml. Plus, it assumes there is a Chart.yaml in the preview folder and replaces the name property. All of that is Helm specific.

Possible solution (breaking change)

• Allow to specify a filename in gitops.config.yaml's previewConfig.replace list. This makes the replacements generic.
• Rely on the user replacing name in the Chart.yaml

Develop with Microsoft Windows

Windows users will have a hard time to build and run this project. Reasons:

• Make is used as build system
• Some locations depending on the host system (such as the directory for temporary files) are expressed as string literals
• Some paths do not use separators of the host system
• …?

Tests don't work with Python 3.10.7

python3 -m pytest -vv -s --typeguard-packages=gitopscli
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pytest/__main__.py", line 7, in <module>
    raise SystemExit(pytest.main())
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 105, in main
    config = _prepareconfig(args, plugins)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 257, in _prepareconfig
    return pluginmanager.hook.pytest_cmdline_parse(
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/helpconfig.py", line 90, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 836, in pytest_cmdline_parse
    self.parse(args)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1044, in parse
    self._preparse(args, addopts=addopts)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 992, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 143, in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
  File "/home/robert/Documents/Baloise/Code/gitopscli/venv/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 330, in _rewrite_test
    co = compile(tree, fn, "exec", dont_inherit=True)
TypeError: required field "lineno" missing from alias
make: *** [Makefile:22: test] Error 1

Deploy enhancements

• provide error handling and usefull log messages and return code when params are not correct
• username / password invalid #49
• org / repo combination not available #49
• git_provider / git_provider_url not valid #49
• yaml file not present at path #41
• values -> some yaml property not present in yaml file #48
• any error while performing git operations
• any error while calling git server api for pr creation, pr merge

Switch to travis-ci.com until December 31st, 2020

travis-ci.org will be closed down December 31st, 2020:
https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom

Fix CI

Travis does not seem to work anymore. See for example #156 which did not trigger any build.

Maybe we could switch to GitHub actions.

Add content to "Getting started" doc

https://baloise-incubator.github.io/gitopscli/getting-started/

Don't fail when deleting a non-existent preview

Currently the gitopscli fails if we want to delete a preview which doesn't exist.
At least, make failing optional (via a flag), e.g.:

1. gitopscli delete-preview --expect-preview-exists <...> -> error
2. gitopscli delete-preview <...> -> nothing to do, i.e. success

Add some documentation

We should have some documentation. At least some more information in the README about what gitopscli is for and what you can do with it.

Improve CLI commands' docs

Add some prose documentation for every CLI command. Also, describe e.g. the required repo setup for some commands.

Use repo's default branch instead of defaulting to master

Currently, there are several commands that can only modify the master branch (e.g. deploy). This should be changed to the default branch of the repository, which still can be master but also supports e.g. more recent Github repositories with their main branches.

Show more info when a push is rejected by git hooks

Some systems use git hooks to enforce policies for a project. Eg, a policy could be to always require a bug tracking issue id in commit messages. If a push is rejected by a git hook it usually displays a message explaining what is wrong with the commit (eg, there was no issue id in the commit message). When pushing with gitopscli the text of a git hook rejection is swallowed. It looks like this:

Error pushing branch 'gitopscli-deploy-c8c10492' to origin.

gitposcli should output as much info as possible about the problem.

Support filters in YAML path

The replace mechanism only supports selecting the YAML path by object key and array index (e.g. foo.bar.[0].baz). This is quite fragile if you have to replace within arrays.

Example:

# kustomizazion.yaml
images:
- name: frontend
  newTag: v2
- name: backend
  newTag: v2

or

# values.yaml
myapp:
   env:
   - name: FOO
     value: foo
   - name: BAR
     value: bar

Replacing the image tags or the env values requires the array to be in a stable order. Also the replacement path is hard to read if you don't know the YAML file: myapp.env.[1].value -> are we replacing FOO or BAR here?

Possible solution:
JsonPath has filter expressions. Selecting the value of FOO would look like this: myapp.env[?(@.name=='FOO')].value

There is a python library jsonpath-ng which can output the matching paths:

>>> jsonpath_expr = parse('foo[*].baz')
# Matches remember where they came from
>>> [str(match.full_path) for match in jsonpath_expr.find({'foo': [{'baz': 1}, {'baz': 2}]})]
['foo.[0].baz', 'foo.[1].baz']

Those returned full_path strings are exactly the simplified YAML path supported by the GitOpsCLI. We could just plug this library in between and translate complicated JsonPath expressions.

The library even has a short form to specify the filters: myapp.env[?name=='FOO'].value

This solution would be backwards compatible. However, we could also choose to just enable this JsonPath translation feature via CLI flag. Any opinions on that?

Is it possible to use with Bitbucket cloud?

Is this posssible to work with bitbucket cloud? As I keep getting 403, like either my repository is not present or I don't have permissions, but I think I should be doing everything right.

So thinking as lot of thing are available for bitbucket server but not cloud, maybe this is the same?

Thanks

Redefining desired state shall not generate an error

Calling the cli with the same version again - i.e. no commit required - generates an error. According to kubernetes philosophy the desired state can be redefined without error.
Requested behaviour: Do not generate an error if no change is being generated.