gitlabform / gitlabform Goto Github PK

View Code? Open in Web Editor NEW

393.0 16.0 87.0 10.17 MB

🏗 Specialized configuration as a code tool for GitLab

Home Page: https://gitlabform.github.io/gitlabform/

License: MIT License

Python 98.61% Dockerfile 0.11% Shell 1.13% Ruby 0.16%

configuration-as-code gitlab yaml cli

gitlabform's People

Stargazers

Watchers

Forkers

sanjusoftware x10an14 beliys davidarcher gitter-badger mmikolajczykm bjarocki pianolinux imouath mrueg dwoods icyfork mkjmdski therealgambo akarem kumy osas1111 serg1i dwinss silverxxx dom-nie kassybas thekinglei srkavin erichhuebner grzesuav rbartuzel pavelkamyshev krystianocado ejiektpobehuk yashwanth-l glensc bradphipps kowpatryk trzeciakp nejch poffey21 pigueiras http425 harry-xm ss7548 amimas mcjffld nvtkaszpir florianludwig gcaaa31928 tmeijn florentio barryib chris-workingmouse cpennycuick aleung trissanen blacktoko tdslinden rflj wrbbz lfvpetber iliashataevarammeem vchepkov l154x morgaroth zanella zhangchunsheng-science benkuhar devinbfergy sachinksingh28 mfriedenhagen speechmatics-bot rajasgujarathi lhokktyn yantantether j4numbers whkslgt tigremoderata nabinbhatt subhadeepbanerjee2003 philippe-granet alamirault timknight-dwp mihkelkillo long-wan-ep oliverbaehler lkwg82 mahadevan-karthi-dwp nudgegoonies luckless

gitlabform's Issues

Handle groups settings

Gitlab has some settings for a group themselves. However, when I try to set up some hook with gitlab-conf, settings are applied for project under the group instead. To be specific, a webhook can be installed at group level, or project level; however specifying hook under groups_settings will deploy the hook for every project under the group.

Relevant source: https://github.com/egnyte/gitlabform/blob/73acba82990655247c8fc5d5be84864ccf792a50/gitlabform/gitlabform/core.py#L210 : there isn't setting for group themselves.

http 401 unauthorized

Hi,

Just trying out this util because of the lack of group deploy keys in Gitlab.

We have a self hosted Gitlab which is exposed using HTTPS.

I am an admin in the system and I created a personal access token with 'api' privileges.

But whenever I try and use gitlabform it errors.

$ gitlabform --debug --noop Puppet
Reading config from: /Users/cliff/config.yml
URL-encoded url=https://gitlab.company.com/api/v3/version
Starting new HTTPS connection (1): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (2): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (3): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (4): gitlab.company.com
Aborting - GitLab test request failed, details: 'HTTPSConnectionPool(host='gitlab.company.com', port=443): Max retries exceeded with url: /api/v3/version (Caused by ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')))'

Branches with "/" in their name are not found when trying to protect/unprotect them

Setting branches
! Branch 'feature/some_branch_name' not found when trying to set it as protected/unprotected

Tag v.1.0.4 contains a dot too much

Can you retag as v1.0.4? Thanks!

Deployment of v.0.17.0 to PyPi fails

See travis-ci/travis-ci#8339

SSL: CERTIFICATE_VERIFY_FAILED

I'm using GitLab CE 12.3.1 with self signed certificate placed under /etc/gitlab/trusted-certs

When I try gitlabform ALL i get:

Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),)': /api/v4/version
Aborting -

Re-running gitlabform with changes to jira services doesn't apply any change

I have noticed that the password variable set in the repo CI/CD variables can't be used in the config.yml, but even when I make changes to my JIRA service settings in the config.yml the jira service remains unchanged. Do you need to manually delete the settings? Or can they be deleted before being re-applied with new settings?

Generally struggling with getting these services set up correctly. I have to replace the password below in order to avoid leaving the password in the config.yml, and replace it at the gitlab-ci.yml level with: sed -i -e 's/SED_REPLACE_JIRA_PASSWORD/'"$JIRA_PASSWORD"'/g' config.yml.

My services configuration, changing the password doesn't seem to get applied, nor does changes to the commit events from true to false.

"group/subgroup/projectname":
  services:
      mattermost:
          active: true
          webhook: https://mattermost.com/hooks/xx9bnr7jijndtgangda64w
          username: gitlab
          merge_requests_events: true
          merge_request_channel: "merge-requests-channel"
          push_events: false
          issues_events: false
          confidential_issues_events: false
          tag_push_events: false
          note_events: false
          confidential_note_events: false
          pipeline_events: false
          wiki_page_events: false
          branches_to_be_notified: "all"
      jira:
          active: true
          url: http://jiralo.xxx.com
          username: "botaccess"
          password: "SED_REPLACE_JIRA_PASSWORD"
          merge_requests_events: true
          commit_events: true

KeyError: 'x-total-pages'

Hello,
first thank you for this great tool,

I got this error:
Connected to GitLab version: 11.9.4-ee (55be7f0)
Config file is declared to be compatible with GitLab API v4
Reading config from: .../config.yml

Processing ALL groups and and projects
Traceback (most recent call last):
File "/home/jo/.local/bin/gitlabform", line 6, in
GitLabForm().main()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlabform/core.py", line 135, in main
projects_and_groups = self.get_projects_list()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlabform/core.py", line 146, in get_projects_list
groups = self.gl.get_groups()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlab/groups.py", line 10, in get_groups
result = self._make_requests_to_api("groups?all_available=true", paginated=True)
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlab/core.py", line 98, in _make_requests_to_api
total_pages = int(first_response.headers['X-Total-Pages'])
File "/home/jo/.local/lib/python3.6/site-packages/requests/structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'x-total-pages'

Create non existent branches

Does it make sense to add the ability to create branches in the tool?

I ask because we want to use it to keep our repo settings in line as people create repos in our namesapce, therefore the ideal flow would be:

repo created > hook fired > gitlabform run > master branch is created and branch protection enabled.

Can't find config.yml

All my setups are according to the documentation, yet, it fails to find the config.yml.

➜  gitlab pwd
/Users/Nomadme/Projects/Code/gitlab
➜  gitlab ll
total 8
drwxr-xr-x   3 Nomadme  staff  102 Jan 29 10:15 ./
drwxr-xr-x  27 Nomadme  staff  918 Jan 29 09:49 ../
-rw-r--r--   1 Nomadme  staff  974 Jan 29 10:14 config.yml
➜  gitlab gitlabform ALL
Aborting - config file not found at: /Users/Nomadme/Projects/Code/gitlab/config.yml

Package for Linux, Mac and maybe Windows

See #30 for rationale.

Implement Push Rules support

https://docs.gitlab.com/ee/api/projects.html#push-rules

Provide some project name and group as built-in variables that you could use in the config

As suggested #6 it would be good to be able to reference the name of currently processed project
and group in its config.

Use case example

Let's assume that you have bunch of custom Puppet modules without metadata.json. You don't need this file for the modules to work, but you need it to start testing them with https://github.com/puppetlabs/pdk and for full Puppet support in some IDEs (like IntelliJ - https://plugins.jetbrains.com/plugin/7180-puppet-support?#comment=24765).

To add this metadata.json with gitlabform, after this issue would be resolved, you could use this as a metadata.json template in files section:

{
  "name": "company-{{ project }}",
  "version": "1.0.0",
  "summary": "{{ project }}",
  "license": "proprietary",
  "source": "https://git.company.com/{{ group }}/{{ project }}",
  "project_page": "https://git.company.com/{{ group }}/{{ project }}/tree/master",
  "dependencies": [
  ],
  "operatingsystem_support": [
    {
      "operatingsystem": "CentOS",
      "operatingsystemrelease": [
        "7"
      ]
    }
  ],
  "requirements": [
    {
      "name": "puppet",
      "version_requirement": ">= 4.7.0 < 6.0.0"
    }
  ],
  "pdk-version": "1.2.0",
  "template-url": "file:///opt/puppetlabs/pdk/share/cache/pdk-module-template.git",
  "template-ref": "heads/master-0-g3f7d5ce"
}

..where all occurrences of {{ project }} would be replaced with current project name and {{ group }} - with group name.

UPDATED: switched to Jinja2-like syntax.

Setting deploy key in some cases fail with HTTP 400: {"fingerprint":["has already been taken"]}

For further investigation:

* [215/420] Processing: internal_apps/watchdog
+++ Error while processing 'internal_apps/watchdog'
Traceback (most recent call last):
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 211, in process_all
    self.process_deploy_keys(project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 31, in method_wrapper
    return method(self, project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 235, in process_deploy_keys
    self.gl.post_deploy_key(project_and_group, configuration['deploy_keys'][deploy_key])
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 40, in post_deploy_key
    self._make_requests_to_api("projects/%s/deploy_keys", pid, 'POST', deploy_key, expected_codes=201)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 47, in _make_requests_to_api
    response = self._make_request_to_api(path_as_format_string, args, method, data, expected_codes)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 83, in _make_request_to_api
    raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request url='https://xxx/api/v3/projects/1144/deploy_keys', method=POST, data='{'key': 'ssh-ed25519 yyyyyyyyyyy [email protected]', 'can_push': True, 'title': '[email protected] auto-bump key'}' failed - expected code(s) 201, got code 400 & body: 'b'{"message":{"fingerprint":["has already been taken"]}}''

Consider migrating to `python-gitlab`

When I started working on Gitlabform (as an internal Egnyte app back then) in the beginning of 2017, one existing GitLab python library I have found had a README that stated something like "Screw it, I am not maintaining this library any longer - GitLab makes too many API changes between even patch versions to keep up with it!"... 🙄

I also remember that some other library I considered was hiding too many GitLab API internals so it was not compatible with the design decision I made to forward values from the YAML config verbatim as parameters for PUTs to GitLab API endpoints (to not have to update Gitlabform app in case of those dreaded API changes that I was warned about above).

Therefore I have started to just use requests and wrote my own library.

I don't remember if I have even found the python-gitlab/python-gitlab and why I have not considered using it...

But now as it has gained friction (1.1k stars as of now) it would be good to stop reinventing the wheel and switch to it. If it lacks something that I need, then I should write a PR to add it to it.

For now I see that is DOES provide both creating new entities by providing raw JSON of parameters as well a direct access to parameters by the same names as in the GitLab API (see examples in https://python-gitlab.readthedocs.io/en/latest/gl_objects/projects.html vs https://docs.gitlab.com/ee/api/projects.html#edit-project) so it seems to be possible relatively easily.

CI / CD schedules

This is more of a feature request. I came across this project while looking for tools that I could use to automatically manage the CI / CD schedules within a GitLab instance. AFAICS gitlabform currently does not support this, but IMHO it would fit in very well.

The API is documented at: https://docs.gitlab.com/ee/api/pipeline_schedules.html

Errors when repo contains periods

Original Name

* [27/218] Processing: foobar-ltd/technology/infra/ans/legacy.ksops.terraform
+++ Error while processing 'foobar-ltd/technology/infra/ans/legacy.ksops.terraform'
Traceback (most recent call last):
  File "/gitlabform/gitlabform/gitlabform/core.py", line 220, in process_all
    self.process_project_push_rules(project_and_group, configuration)
  File "/gitlabform/gitlabform/gitlabform/core.py", line 31, in method_wrapper
    return method(self, project_and_group, configuration)
  File "/gitlabform/gitlabform/gitlabform/core.py", line 248, in process_project_push_rules
    self.gl.put_project_push_rules(project_and_group, push_rules)
  File "/gitlabform/gitlabform/gitlab/projects.py", line 90, in put_project_push_rules
    self._make_requests_to_api("projects/%s/push_rule", project_and_group_name, 'PUT', push_rules)
  File "/gitlabform/gitlabform/gitlab/core.py", line 83, in _make_requests_to_api
    response = self._make_request_to_api(path_as_format_string, args, method, data, expected_codes, json)
  File "/gitlabform/gitlabform/gitlab/core.py", line 116, in _make_request_to_api
    raise NotFoundException("Resource path='%s' not found!" % url)
gitlabform.gitlab.core.NotFoundException: Resource path='https://gitlab.com/api/v4/projects/foobar-ltd%2Ftechnology%2Finfra%2Fans%2Flegacy.ksops.terraform/push_rule' not found!

After removing . from the name

* [8/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-base
* [9/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-credentials
* [10/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-iamfit
* [11/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-iamsplunk
* [12/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-puppetmaster
* [13/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-terraform

➜ gitlab-config

Write a doc about contributing (with architecture, design and a code example)

I feel that if I document how this app is designed and provide a real-case example of how to add features to it, it will get much more PRs and with community help will improve much faster than I can do it on my own.

Working with groups with spaces

Currently you don't seem to be able to operate against groups with spaces.

Protected branches API not implemented?

Hi,

I read the documentation up and down but could not find how to do what described here https://docs.gitlab.com/ee/api/protected_branches.html#protect-repository-branches. Is this not implemented yet? Thanx

Allow setting file contents from external files, not only embedded in the config.yml

We have got to a point in our internal app use that we have a lot of files embedded in the config in the files section.

Therefore I would to add a feature to enable passing paths to files instead of only their contents.

Of course the current way of passing files contents also makes sense for some use cases so it will be kept.

Show what will really change

Hi.

It would be great if gitlabform could show what will change before applying (i.e. the difference between the current state and configured state).

I was thinking about something like that:

Resource actions are indicated with the following symbols:
  - delete
  + create
  ~ update in-place

Configuration that would be applied:

~ project1
    project_settings:
        visibility:     "internal" => "private"

~ project2
    project_settings:
        visibility:     "internal" => "private"

This is inspired by the output of terraform.

Question: Removing keys

Is it possible to remove keys from project/group/common?

Gitlab 10.3 compatibility

Is there any plans to support Gitlab 10.3?

handling secrets with gitlabform

GitlabForm can be succesfully used to provision Gitlab projects with secrets, like passwords, API keys etc.

This is extremely helpful. It also means that with the current design, those secrets need to be hardcoded inside the config.yml file

It be great if sensitive data could be passed from external sources. Reasons:

security; developers in the company may be privy to seeing general configuration of Gitlab so as a reference, so that they know what to expect in a specific project.. yet at the same time, secrets shouldn't be that easily readable
single source of truth: users of GitlabForm may be keeping their secrets in various safe places and apps: Vault, LastPass, other password managers, even in an encrypted text file or a sticky note (in a safe, obviously :) )
additionallly (nice to have), the secrets shouldn't be visible in plain text in GitlabForm logs

At the moment the only workaround for that that I can think of would be to keep have config.yml file as a template to be filled in at runtime (e.g. with https://pypi.org/project/j2cli/ or https://pypi.org/project/jinja2-vault/). This however:

complicates the flow as you'd need to re-generate the config each time you want to apply it, lint it (see https://github.com/adrienverge/yamllint)
may interfere with the current templating capabilities (re {{ project }} / {{ group }}).

Do you ( @gdubicki - project owner) have any plans (or even just opinions) on such functionality?

Breaking changes proposals for GitLabForm 2.*

As explained in this comment we plan to uptick GitLabForm major version to 2 because we will change the config syntax.

We will do that together for #27 and #68 but while doing that we can also improve config syntax in some other ways that are not possible in v. 1 as they would be backward incompatible.

And we can also consider other changes, which would be breaking ones.

Everyone is welcome to share your opinions and ideas. :) I am pinging some specific people as recently involved users / contributors, but it does not mean that we are not listening to other opinions. :)

Users active in recent issues: @weakcamel, @Dridge, @smcgregor83, @masonbivens, @ginkel ?

Contributors: @mkjmdski, @akaRem, @SilverXXX, @srkavin, @mrueg, @icy, @dom-nie, @davidarcher, @brtj, @johanfleury, @kassybas

installation problem on MacOS due to fixed PyYaml version

When installing gitlabform on MacOS (High Sierra), I'm facing problems.

As it happens, the specified version (3.12) is affected by problems described in yaml/pyyaml#15

therefore gitlabform fails to install with pip as well.

The solution seems to be simple - update the dependency to pyyaml 3.13 which solves the problem, or don't requre a specific version in the first place.

Only process projects from within group namespace

GitLab's API for getting projects within a project group (https://docs.gitlab.com/ce/api/groups.html#list-a-group-39-s-projects) returns also projects which are not really in the project group, but for which this group is used as a member.

For example let's assume that you have a project group foo with a project foo/bar in it. You also have a project different/other in a different group. And you make foo a member of the different/other project.

Then when you get projects from group foo you will get both foo/bar and different/other...

This is problematic for gitlabform as at least when we use it and configure it to apply some config for projects within a group foo then you mean to apply it to all projects within that group and only in that group namespace (so in the example above: only foo/bar)...

For now we used skip_projects as a workaround for this, but this is too problematic at this point to keep doing that.

Detect configuration syntax errors

Because of the "raw parameters passing" design concept applied in this app you cannot detect all kinds of syntax errors with a reasonable amount of development effort until you try to apply the config.

But we should be able to detect some key syntax problems to prevent users from having doubts like mentioned here:

I could just be using it wrong (although I've tried moving the key "approvers" under "approvals" and this didn't work either).

Let's fix this! Ideas on how to do it are very welcome. :) I am thinking about generating the effective config and then checking it against some YAML-schema thing if something like that exists.

Exit when the first error occurs

Is it possible to terminate the program on the first error? That would be very useful when using the tool as a CI tool for gitlab per-se.

Current behavior: The program continues event if some error occurs (https://github.com/egnyte/gitlabform/blob/84b7986aacb8417021ed8ea7ee2efcc3e517506b/gitlabform/gitlabform/core.py#L222)

Expected behavior: The program terminates immediately when the first error occurs.

Error with projects under certain group namespace

I'm receiving the following error when trying to apply against a newly created repo.

➜ gitlab-config git:(master!) gitlabform -c config.yml ALL_DEFINED
>>> Processing ALL groups and projects defined in config
*** # of groups to process: 50
*** # of projects to process: 18
* [1/18] Processing: company/technology/dev/foobar/foobar-ab-library
* [2/18] Processing: company/technology/dev/foobar/foobar-admin-gateway
* [3/18] Processing: company/technology/dev/foobar/foobar-enrollment
* [4/18] Processing: company/technology/dev/foobar/foobar-fi-config
* [5/18] Processing: company/technology/dev/foobar/foobar-fi-config-web
* [6/18] Processing: company/technology/dev/foobar/foobar-gateway
* [7/18] Processing: company/technology/dev/foobar/foobar-mobile-app
* [8/18] Processing: company/technology/dev/foobar/foobar-profile
* [9/18] Processing: company/technology/dev/foobar/pfm_demo
* [10/18] Processing: company/technology/dev/pi/test
Traceback (most recent call last):
  File "/usr/local/bin/gitlabform", line 6, in <module>
    GitLabForm().main()
  File "/usr/local/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 136, in main
    self.process_all(projects_and_groups)
  File "/usr/local/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 201, in process_all
    configuration = self.c.get_effective_config_for_project(project_and_group)
  File "/usr/local/lib/python3.7/site-packages/gitlabform/configuration/projects_and_groups.py", line 54, in get_effective_config_for_project
    for key in group_config.keys() | common_config.keys():
AttributeError: 'NoneType' object has no attribute 'keys'

Deleting the repo or commenting out that group in the config allows for a successful run.

CONFIG

# GitLab API access config
gitlab:
  url: https://gitlab.com
  token: ""
  api_version: 4

#
# You can define settings on 3 levels here:
# * common - for ALL projects in ALL groups
# * group - for ALL projects in selected groups
# * project - for single projects
#
# Exception: when `skip: true` is set on a more specific level, given config section is not set AT ALL for given project.
#

common_settings:

  project_settings:
    jobs_enabled: true
    public_jobs: false
    visibility: private
    default_branch: master
    issues_enabled: false
    snippets_enabled: false
    wiki_enabled: false
    shared_runners_enabled: false
    only_allow_merge_if_pipeline_succeeds: true
    only_allow_merge_if_all_discussions_are_resolved: false

  merge_requests:
    approvals:
      approvals_before_merge: 2
      reset_approvals_on_push: false
      disable_overriding_approvers_per_merge_request: true

  project_push_rules:
    prevent_secrets: true
    author_email_regex: ""

  branches:
    master:
      protected: true
      developers_can_push: false
      developers_can_merge: true


group_settings:
############################################
#     Developer Related Repos
############################################
  "company/technology/dev/an":
  "company/technology/dev/apt":
  "company/technology/dev/auto":
  "company/technology/dev/bcm":
  "company/technology/dev/cdn":
  "company/technology/dev/chat":
  "company/technology/dev/cms":
  "company/technology/dev/com":
  "company/technology/dev/crm":
  "company/technology/dev/dep":
  "company/technology/dev/dq":
  "company/technology/dev/dw":
  "company/technology/dev/edw":
  "company/technology/dev/fed":
  "company/technology/dev/fid":
  "company/technology/dev/gen2":
  "company/technology/dev/gov":
  "company/technology/dev/hack":
  "company/technology/dev/hub":
  "company/technology/dev/kcas":
    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

  "company/technology/dev/kloan":
  "company/technology/dev/lamp":
  "company/technology/dev/lemp":
  "company/technology/dev/llib":
  "company/technology/dev/map":
  "company/technology/dev/ms":
  "company/technology/dev/op":
  "company/technology/dev/pam":
  "company/technology/dev/pdm":
  "company/technology/dev/phpv":
  "company/technology/dev/pi":
  "company/technology/dev/pmt":
  "company/technology/dev/poc":
  "company/technology/dev/ppm":
  "company/technology/dev/pres":
  "company/technology/dev/rct":
  "company/technology/dev/rcx":
  "company/technology/dev/res":
  "company/technology/dev/seed":
  "company/technology/dev/stat":
  "company/technology/dev/sui":
  "company/technology/dev/svc":
  "company/technology/dev/tran":

############################################
#     Infrastructure Related Repos
############################################
# Packer Projects
  "company/technology/infra/ami":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false


  "company/technology/infra/ans":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

  "company/technology/infra/aws":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

  "company/technology/infra/cfg":

  "company/technology/infra/chef":

  "company/technology/infra/cloud-ops":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

  "company/technology/infra/doc":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false


"company/technology/infra/dpk":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false


"company/technology/infra/glci":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

"company/technology/infra/helm":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

"company/technology/infra/pup":


"company/technology/infra/run":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

"company/technology/infra/sys":

"company/technology/infra/ter":

    services:
      slack:
        webhook: ""
        username: "Gitlab Bot"
        channel: ""
        notify_only_default_branch: true
        confidential_issues_events: false
        issues_events: false
        merge_requests_events: true
        note_events: false
        pipeline_events: true
        push_events: false
        tag_push_events: false
        wiki_page_events: false

"company/technology/infra/tut":

Hierarchical .gitlab-ci.yml file management

(Perhaps it's out of the scope of this app and should be done with some templating tool after #5 is implemented, but I would like to describe the idea here anyway.)

It would be nice to be able to manage .gitlab-ci.yml files in an additive way.

For example we have some tests that are shared for all repos in group X, but would like to add some additional ones only in projects X/p1 and X/p2.

Other example is that we do deployments in all repos in group X the same way, except that we don't do them at all for project X/p3.

I am thinking about using a new config section, let's say 'gitlab_ci_yaml' (this name is on purpose different that the filename so that it won't be confused with files: entry), that could accept .gitlab-ci.yml syntax extended with some templating features, like:

extra key "gitlabform_skip: true" would make given element not added to the final file,
extra key "gitlabform_replace: true" would make given element replace with an element of the same name found on a higher level,
extra key "gitlabform_merge: true" would make given element merge with an element of the same name found on a higher level

I would not use either merging or replacing by default, because the default is non-obvious and expect one of the 2 last keys to be present if you redefine some element, like a job, on a lower

Example config:

group_settings:
  X:
    gitlab_ci_yaml:
      branches: all
      yaml:
        variables:
          THING_A: some_value

        stages:
          - test
          - deploy

        test:
          stage: test
          script:
            - ./run_test.sh

        deployment:
          stage: deploy
          script:
            - ./deploy.sh

project_settings:
  X/p1:
    gitlab_ci_yaml:
      branches: all
      yaml:
        test_something_p1_specific:
          stage: test
          script:
            - ./run_test_p1.sh

       deployment:
         gitlabform_replace: true
         stage: deploy
         script:
          - ./deploy_p1.sh

Effective .gitlab-ci.yml in all projects in X group except p1:

variables:
  THING_A: some_value

stages:
  - test
  - deploy

test:
  stage: test
  script:
    - ./run_test.sh

deployment:
  stage: deploy
  script:
    - ./deploy.sh

..and in X/p1:

variables:
  THING_A: some_value

stages:
  - test
  - deploy

test:
  stage: test
  script:
    - ./run_test.sh

test_something_p1_specific:
  stage: test
  script:
    - ./run_test_p1.sh

deployment:
  stage: deploy
  script:
    - ./deploy_p1.sh

If anyone reads this :P then please let me know what you think about this proposal.

Some services always apply certain events as true

When adjusting the Jira service the commit_events option is always true, and has to be set to false manually through the UI, the same is true for the mattermost service and the confidential_issues_events option.

project_settings:
    "group/subgroup/project":
      services:
          mattermost:
              active: true
              webhook: https://mattermost.com/hooks/xxx
              username: gitlab
              merge_requests_events: true
              merge_request_channel: "merge-requests"
              push_events: false
              issues_events: false
              confidential_issues_events: false # This doesnt appear to work, is always true, in all cases
              tag_push_events: false
              note_events: false
              confidential_note_events: false
              pipeline_events: false
              wiki_page_events: false
              branches_to_be_notified: "all"
          jira:
              active: true
              url: http://jiralo.exfo.com
              username: "user1"
              password: "password"
              merge_requests_events: true
              commit_events: false # This doesnt appear to work, is always true, in all cases

Can it do these things:

Looks really cool, had a quick try with it and there's a few things I'm not able to get working, wondering if they can be done and if yes how:

set group visibility
apply configuration to projects in subgroups recursively
create group
create project
use project name (and potentially other information gathered) as a parameter in config strings (usecase here is I need to create a hook that has project name in the url for every project)

Configure Slack notifications

How do I configure Slack notifications?

services:
  slack:
    webhook: "url"
    username: "username"

Support multiple config files

It may be good to read more than one config file to have more smaller files with configs for different groups rather than one big file with everything.

But to avoid the effective config being hard(er) to understand I would NOT support having config for a single group/project spread in more than one file.

Trying to set secret variables if CI is not set up in a project fails

If project does not have CI set up (= have .gitlab-ci.yml in its default branch?) then setting secret variable on it fails on getting current secret variables values:

gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/XXX/variables', method=GET, data='None' failed - expected code(s) 200, got code 403 & body: 'b'{"message":"403 Forbidden"}''

global variables for ci builds

Sourced from https://gitlab.com/gitlab-org/gitlab-ce/issues/3897.

Settings hooks fails sometimes

I'll just leave it here for now so in case anyone else will encounter this will know that this is a known issue.

Please add thumbs up to this issue if you have this problem too!

* [359/407] Processing: sddc/openstack-corosync-pacemaker
+++ Error while processing 'sddc/openstack-corosync-pacemaker'
Traceback (most recent call last):
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 189, in process_all
    self.process_hooks(project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 30, in method_wrapper
    return method(self, project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 65, in method_wrapper
    return method(self, project_and_group, SafeDict(configuration))
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 364, in process_hooks
    self.gl.post_hook(project_and_group, hook, configuration['hooks'][hook])
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 96, in post_hook
    self._make_requests_to_api("projects/%s/hooks" % pid, 'POST', data, expected_codes=201)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 46, in _make_requests_to_api
    response = self._make_request_to_api(path, method, data, expected_codes)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 81, in _make_request_to_api
    raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/1780/hooks', method=POST, data='{'push_events': False, 'merge_requests_events': True, 'url': 'http://127.0.0.1:5000/hooks/merge-request'}' failed - expected code(s) 201, got code 500 & body: 'b'{"message":"500 Internal Server Error"}''
* [360/407] Processing: sddc/openstack-corosync-pacemaker-wrapper
! Branch 'master' not found when trying to set it as protected/unprotected
+++ Error while processing 'sddc/openstack-corosync-pacemaker-wrapper'
Traceback (most recent call last):
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 189, in process_all
    self.process_hooks(project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 30, in method_wrapper
    return method(self, project_and_group, configuration)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 65, in method_wrapper
    return method(self, project_and_group, SafeDict(configuration))
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 364, in process_hooks
    self.gl.post_hook(project_and_group, hook, configuration['hooks'][hook])
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 96, in post_hook
    self._make_requests_to_api("projects/%s/hooks" % pid, 'POST', data, expected_codes=201)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 46, in _make_requests_to_api
    response = self._make_request_to_api(path, method, data, expected_codes)
  File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 81, in _make_request_to_api
    raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/1781/hooks', method=POST, data='{'push_events': False, 'merge_requests_events': True, 'url': 'http://127.0.0.1:5000/hooks/merge-request'}' failed - expected code(s) 201, got code 500 & body: 'b'{"message":"500 Internal Server Error"}'"

Migrate to API v4

This issue is here to let you know that we are commited to maintaining this project and I will make this migration soon. So you don't have to worry that gitlabform will stop working with subsequent GitLab versions. :) Also please note that API v3 did not go away in 9.5 (http://disq.us/p/1lky6q4) and will not go away soon, so there is no rush with this. See https://gitlab.com/gitlab-org/gitlab-ce/issues/36819 for details.

Implement badges endpoint

It's a small thing but I would find this useful. E.g. I want to add a badge so people know the project is managed by gitlabform.

Project badges are managed under this endpoint https://docs.gitlab.com/ee/api/project_badges.html
Group badges use this one https://docs.gitlab.com/ee/api/group_badges.html

You can't enable builds for private projects

Because of GitLab bug https://gitlab.com/gitlab-org/gitlab-ce/issues/27298 you can't use the following config:

my_group/my_project:
  project_settings:
    builds_enabled: true
    visibility_level: 0

because it results in HTTP 400 response from GitLab API:

{
  'project_feature.builds_access_level': [
    'cannot have higher visibility level than repository access level'
  ]
}

This is of the long standing bugs that was one of the reasons we implemented project skipping: https://github.com/egnyte/gitlabform/blob/master/config.yml#L119

But we can no longer skip such projects so I plan to implement a better workaround that will allow skipping only a part of project config (such as project_settings, deploy_keys etc.), even if it is configured on a group level (like we do for files: https://github.com/egnyte/gitlabform/blob/master/config.yml#L117 ).

1.8.0 version issues

Hello,
I've tried upgrading from v1.6.0 to 1.8.0 and encountered problems with it.

Stack trace:

*** # of groups to process: 39
*** # of projects to process: 238
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/config-Z17lvQEG/bin/gitlabform", line 6, in <module>
    GitLabForm().main()
  File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 135, in main
    self.process_all(projects_and_groups, groups)
  File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 197, in process_all
    configuration = self.c.get_effective_config_for_group(group)
  File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/configuration/projects_and_groups.py", line 111, in get_effective_config_for_group
    group_config = self.get_config_for_group(group)
AttributeError: 'Configuration' object has no attribute 'get_config_for_group'

Changing files in branches protected manually (not defined in gitlabform config) fail

Because gitlabform only does the unprotect branch -> change files -> protect branch again for branches which are configured as protected in its config, it fails to change files in branches protected manually.

I am not sure if this should actually be fixed though, as you should not configure protected branches manually if you also use gitlabform to configure the project...

Changing files in a lot of files, in many branches, in many projects can make GitLab unresponsive

In our GitLab instance in some of our groups managed with gitlabform, we have almost 100 projects, sometimes with more than 100 branches.

Because of that if we change a few files by config files: section change (.gitlab-ci.yml and some related files) we create A LOT of new pipelines in GitLab within just seconds during gitlabform run.

Even with automatic pipeline cancellation this has caused our GitLab instance to become unresponsive and start returning 503 for subsequent API calls.

I have thought about making changes in files within a single branch with a single commit but as we would have to switch from using https://docs.gitlab.com/ee/api/repository_files.html#update-existing-file-in-repository to commandline git, it would be quite hard.

Also the benefits would not be big now that we have auto cancelling redundant pipelines in GitLab, since 9.1 IIRC.

Therefore as a workaround for this issue I would wrap changing files stage with disabling pipelines and re-enabling them afterwards.

(I thought about some way to disable writes to this repo in the mean time to minimize possibility that someone will push their commits into the repo while pipelines are disabled, but I can't use archiving feature because that would prevent gitlabform from editing the files too and I don't want to start messing with project membership at this point.)

Manage files in protected branches only

In files section, for branches key we now support array of branch names or a string "all".

It would be useful to be able to also use a string "protected" that would make the file be managed only in protected branches.

MR approvers user/group has no effect (REST API change?)

Applying a configuration using approvers and approver_groups has no effect while running against Gitlab 12.3.5-ee.

The config I'm using:

group_settings:
  'very-silly-test-group-for-ldap-and-fun':
    merge_requests:
      approvals:
        approvals_before_merge: 1
        reset_approvals_on_push: false
        disable_overriding_approvers_per_merge_request: false
        merge_requests_author_approval: true
      approver_groups:
        - gitlab
      approvers:
        - waldekm
        - foouser

With the log

$ gitlabform -vvv -c config.yml very-silly-test-group-for-ldap-and-fun
Reading config from: /Users/waldekm/git/config/config.yml
Connected to GitLab version: 12.3.5-ee (9dbaa740018)
Config file is declared to be compatible with GitLab API v4
Reading config from: /Users/waldekm/git/config/config.yml
*** # of groups to process: 1
*** # of projects to process: 1
* [1/1] Processing: very-silly-test-group-for-ldap-and-fun/testy-project
Setting project_settings
Setting project settings: {'only_allow_merge_if_pipeline_succeeds': True, 'only_allow_merge_if_all_discussions_are_resolved': True, 'shared_runners_enabled': True, 'request_access_enabled': True, 'wiki_enabled': False, 'snippets_enabled': False}
Setting merge_requests
Setting approvals settings: {'approvals_before_merge': 1, 'reset_approvals_on_push': False, 'disable_overriding_approvers_per_merge_request': False, 'merge_requests_author_approval': True}
Setting approvers to users ['waldekm', 'foouser'] and groups ['gitlab']
Setting hooks

Gitlabform implementation uses endpoint projects/:id/approvers which has been deprecated:
https://docs.gitlab.com/ee/api/merge_request_approvals.html#change-allowed-approvers

I could just be using it wrong (although I've tried moving the key approvers under approvals and this didn't work either). It might be also that the deprecated endpoint doesn't work anymore. In any case, adding/removing users to the approvers list and running gitlabform doesn't show in gitlab-rails/api_json.log

Support for project-level approvals (Gitlab 10.6 EE)

Hi guys,

Would it be possible to add the following resource to the tool? POST /projects/:id/approvals

Gitlab API Documentation

Potential problem: It is only available to EE starter and above. Not sure how you handle this in the tool...

Let me know if I can be of any help! I am not very familiar with Python, but I'm sure I could make something work.

Cheers!

Add ~integration~ acceptance tests using GitLab instance in Docker

Lack of tests in this app is something I am ashamed of for a long time now. I used to excuse myself by stating that running Gitlabform at Egnyte for >800 repositories every day in our CI is some way of testing it.

And partially it is but it does not detect all issues.

Some issues are related to things that used to work stop working and we do not notice that because we did not reconfigure repos to a different config to notice lack of change. This is is the case of not noticing issues like #68.

Other issues are problems related to apparent GitLab API regressions. This is the case of #69.

Therefore I would like to add ~~integration~~ acceptance tests that will:

spin up GitLab instance in a Docker container and provide a basic set up,
for each feature of Gitlabform test if it works to: 1) enable it, 2) reconfigure it, 3) disable it.

Apart from running them after each commit I want to set up a schedul to run it every day or every week to detect problems caused by changes in the new GitLab versions.

UPDATE: I was pointed out that these are not integrations tests but rather acceptance tests. Thanks for that!

Implement the "protected branches" API endpoint

Hey there!

That endpoint would be pretty useful for us, as we want to have a branch on a project that should not be mergeable by anybody...

Sadly, I couldn't find a reference to it in the code. Are you planning on implementing it?

Cheers!

Gitlab Documentation: https://docs.gitlab.com/ee/api/protected_branches.html#protect-repository-branches