gitlabform / gitlabform Goto Github PK
View Code? Open in Web Editor NEW๐ Specialized configuration as a code tool for GitLab
Home Page: https://gitlabform.github.io/gitlabform/
License: MIT License
๐ Specialized configuration as a code tool for GitLab
Home Page: https://gitlabform.github.io/gitlabform/
License: MIT License
Sourced from https://gitlab.com/gitlab-org/gitlab-ce/issues/3897.
I'll just leave it here for now so in case anyone else will encounter this will know that this is a known issue.
Please add thumbs up to this issue if you have this problem too!
* [359/407] Processing: sddc/openstack-corosync-pacemaker
+++ Error while processing 'sddc/openstack-corosync-pacemaker'
Traceback (most recent call last):
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 189, in process_all
self.process_hooks(project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 30, in method_wrapper
return method(self, project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 65, in method_wrapper
return method(self, project_and_group, SafeDict(configuration))
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 364, in process_hooks
self.gl.post_hook(project_and_group, hook, configuration['hooks'][hook])
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 96, in post_hook
self._make_requests_to_api("projects/%s/hooks" % pid, 'POST', data, expected_codes=201)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 46, in _make_requests_to_api
response = self._make_request_to_api(path, method, data, expected_codes)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 81, in _make_request_to_api
raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/1780/hooks', method=POST, data='{'push_events': False, 'merge_requests_events': True, 'url': 'http://127.0.0.1:5000/hooks/merge-request'}' failed - expected code(s) 201, got code 500 & body: 'b'{"message":"500 Internal Server Error"}''
* [360/407] Processing: sddc/openstack-corosync-pacemaker-wrapper
! Branch 'master' not found when trying to set it as protected/unprotected
+++ Error while processing 'sddc/openstack-corosync-pacemaker-wrapper'
Traceback (most recent call last):
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 189, in process_all
self.process_hooks(project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 30, in method_wrapper
return method(self, project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 65, in method_wrapper
return method(self, project_and_group, SafeDict(configuration))
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 364, in process_hooks
self.gl.post_hook(project_and_group, hook, configuration['hooks'][hook])
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 96, in post_hook
self._make_requests_to_api("projects/%s/hooks" % pid, 'POST', data, expected_codes=201)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 46, in _make_requests_to_api
response = self._make_request_to_api(path, method, data, expected_codes)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 81, in _make_request_to_api
raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/1781/hooks', method=POST, data='{'push_events': False, 'merge_requests_events': True, 'url': 'http://127.0.0.1:5000/hooks/merge-request'}' failed - expected code(s) 201, got code 500 & body: 'b'{"message":"500 Internal Server Error"}'"
Is there any plans to support Gitlab 10.3?
I'm using GitLab CE 12.3.1 with self signed certificate placed under /etc/gitlab/trusted-certs
When I try gitlabform ALL
i get:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),)': /api/v4/version
Aborting -
This issue is here to let you know that we are commited to maintaining this project and I will make this migration soon. So you don't have to worry that gitlabform will stop working with subsequent GitLab versions. :) Also please note that API v3 did not go away in 9.5 (http://disq.us/p/1lky6q4) and will not go away soon, so there is no rush with this. See https://gitlab.com/gitlab-org/gitlab-ce/issues/36819 for details.
Because gitlabform only does the unprotect branch -> change files -> protect branch again for branches which are configured as protected in its config, it fails to change files in branches protected manually.
I am not sure if this should actually be fixed though, as you should not configure protected branches manually if you also use gitlabform to configure the project...
Hey there!
That endpoint would be pretty useful for us, as we want to have a branch on a project that should not be mergeable by anybody...
Sadly, I couldn't find a reference to it in the code. Are you planning on implementing it?
Cheers!
Gitlab Documentation: https://docs.gitlab.com/ee/api/protected_branches.html#protect-repository-branches
As suggested #6 it would be good to be able to reference the name of currently processed project
and group in its config.
Let's assume that you have bunch of custom Puppet modules without metadata.json
. You don't need this file for the modules to work, but you need it to start testing them with https://github.com/puppetlabs/pdk and for full Puppet support in some IDEs (like IntelliJ - https://plugins.jetbrains.com/plugin/7180-puppet-support?#comment=24765).
To add this metadata.json
with gitlabform, after this issue would be resolved, you could use this as a metadata.json
template in files
section:
{
"name": "company-{{ project }}",
"version": "1.0.0",
"summary": "{{ project }}",
"license": "proprietary",
"source": "https://git.company.com/{{ group }}/{{ project }}",
"project_page": "https://git.company.com/{{ group }}/{{ project }}/tree/master",
"dependencies": [
],
"operatingsystem_support": [
{
"operatingsystem": "CentOS",
"operatingsystemrelease": [
"7"
]
}
],
"requirements": [
{
"name": "puppet",
"version_requirement": ">= 4.7.0 < 6.0.0"
}
],
"pdk-version": "1.2.0",
"template-url": "file:///opt/puppetlabs/pdk/share/cache/pdk-module-template.git",
"template-ref": "heads/master-0-g3f7d5ce"
}
..where all occurrences of {{ project }}
would be replaced with current project name and {{ group }}
- with group name.
UPDATED: switched to Jinja2-like syntax.
GitlabForm can be succesfully used to provision Gitlab projects with secrets, like passwords, API keys etc.
This is extremely helpful. It also means that with the current design, those secrets need to be hardcoded inside the config.yml
file
It be great if sensitive data could be passed from external sources. Reasons:
At the moment the only workaround for that that I can think of would be to keep have config.yml
file as a template to be filled in at runtime (e.g. with https://pypi.org/project/j2cli/ or https://pypi.org/project/jinja2-vault/). This however:
{{ project }}
/ {{ group }}
).Do you ( @gdubicki - project owner) have any plans (or even just opinions) on such functionality?
Hi.
It would be great if gitlabform
could show what will change before applying (i.e. the difference between the current state and configured state).
I was thinking about something like that:
Resource actions are indicated with the following symbols:
- delete
+ create
~ update in-place
Configuration that would be applied:
~ project1
project_settings:
visibility: "internal" => "private"
~ project2
project_settings:
visibility: "internal" => "private"
This is inspired by the output of terraform.
Because of GitLab bug https://gitlab.com/gitlab-org/gitlab-ce/issues/27298 you can't use the following config:
my_group/my_project:
project_settings:
builds_enabled: true
visibility_level: 0
because it results in HTTP 400 response from GitLab API:
{
'project_feature.builds_access_level': [
'cannot have higher visibility level than repository access level'
]
}
This is of the long standing bugs that was one of the reasons we implemented project skipping: https://github.com/egnyte/gitlabform/blob/master/config.yml#L119
But we can no longer skip such projects so I plan to implement a better workaround that will allow skipping only a part of project config (such as project_settings
, deploy_keys
etc.), even if it is configured on a group level (like we do for files: https://github.com/egnyte/gitlabform/blob/master/config.yml#L117 ).
How do I configure Slack notifications?
services:
slack:
webhook: "url"
username: "username"
We have got to a point in our internal app use that we have a lot of files embedded in the config in the files
section.
Therefore I would to add a feature to enable passing paths to files instead of only their contents.
Of course the current way of passing files contents also makes sense for some use cases so it will be kept.
If project does not have CI set up (= have .gitlab-ci.yml
in its default branch?) then setting secret variable on it fails on getting current secret variables values:
gitlabform.gitlab.core.UnexpectedResponseException: Request path='projects/XXX/variables', method=GET, data='None' failed - expected code(s) 200, got code 403 & body: 'b'{"message":"403 Forbidden"}''
Original Name
* [27/218] Processing: foobar-ltd/technology/infra/ans/legacy.ksops.terraform
+++ Error while processing 'foobar-ltd/technology/infra/ans/legacy.ksops.terraform'
Traceback (most recent call last):
File "/gitlabform/gitlabform/gitlabform/core.py", line 220, in process_all
self.process_project_push_rules(project_and_group, configuration)
File "/gitlabform/gitlabform/gitlabform/core.py", line 31, in method_wrapper
return method(self, project_and_group, configuration)
File "/gitlabform/gitlabform/gitlabform/core.py", line 248, in process_project_push_rules
self.gl.put_project_push_rules(project_and_group, push_rules)
File "/gitlabform/gitlabform/gitlab/projects.py", line 90, in put_project_push_rules
self._make_requests_to_api("projects/%s/push_rule", project_and_group_name, 'PUT', push_rules)
File "/gitlabform/gitlabform/gitlab/core.py", line 83, in _make_requests_to_api
response = self._make_request_to_api(path_as_format_string, args, method, data, expected_codes, json)
File "/gitlabform/gitlabform/gitlab/core.py", line 116, in _make_request_to_api
raise NotFoundException("Resource path='%s' not found!" % url)
gitlabform.gitlab.core.NotFoundException: Resource path='https://gitlab.com/api/v4/projects/foobar-ltd%2Ftechnology%2Finfra%2Fans%2Flegacy.ksops.terraform/push_rule' not found!
After removing .
from the name
* [8/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-base
* [9/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-credentials
* [10/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-iamfit
* [11/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-iamsplunk
* [12/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-puppetmaster
* [13/13] Processing: foobar-ltd/technology/infra/ans/legacy-ksops-terraform
โ gitlab-config
Looks really cool, had a quick try with it and there's a few things I'm not able to get working, wondering if they can be done and if yes how:
It's a small thing but I would find this useful. E.g. I want to add a badge so people know the project is managed by gitlabform.
Project badges are managed under this endpoint https://docs.gitlab.com/ee/api/project_badges.html
Group badges use this one https://docs.gitlab.com/ee/api/group_badges.html
It may be good to read more than one config file to have more smaller files with configs for different groups rather than one big file with everything.
But to avoid the effective config being hard(er) to understand I would NOT support having config for a single group/project spread in more than one file.
Hello,
first thank you for this great tool,
I got this error:
Connected to GitLab version: 11.9.4-ee (55be7f0)
Config file is declared to be compatible with GitLab API v4
Reading config from: .../config.yml
Processing ALL groups and and projects
Traceback (most recent call last):
File "/home/jo/.local/bin/gitlabform", line 6, in
GitLabForm().main()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlabform/core.py", line 135, in main
projects_and_groups = self.get_projects_list()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlabform/core.py", line 146, in get_projects_list
groups = self.gl.get_groups()
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlab/groups.py", line 10, in get_groups
result = self._make_requests_to_api("groups?all_available=true", paginated=True)
File "/home/jo/.local/lib/python3.6/site-packages/gitlabform/gitlab/core.py", line 98, in _make_requests_to_api
total_pages = int(first_response.headers['X-Total-Pages'])
File "/home/jo/.local/lib/python3.6/site-packages/requests/structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'x-total-pages'
As explained in this comment we plan to uptick GitLabForm major version to 2 because we will change the config syntax.
We will do that together for #27 and #68 but while doing that we can also improve config syntax in some other ways that are not possible in v. 1 as they would be backward incompatible.
And we can also consider other changes, which would be breaking ones.
Everyone is welcome to share your opinions and ideas. :) I am pinging some specific people as recently involved users / contributors, but it does not mean that we are not listening to other opinions. :)
Users active in recent issues: @weakcamel, @Dridge, @smcgregor83, @masonbivens, @ginkel ?
Contributors: @mkjmdski, @akaRem, @SilverXXX, @srkavin, @mrueg, @icy, @dom-nie, @davidarcher, @brtj, @johanfleury, @kassybas
GitLab's API for getting projects within a project group (https://docs.gitlab.com/ce/api/groups.html#list-a-group-39-s-projects) returns also projects which are not really in the project group, but for which this group is used as a member.
For example let's assume that you have a project group foo
with a project foo/bar
in it. You also have a project different/other
in a different
group. And you make foo
a member of the different/other
project.
Then when you get projects from group foo
you will get both foo/bar
and different/other
...
This is problematic for gitlabform
as at least when we use it and configure it to apply some config for projects within a group foo
then you mean to apply it to all projects within that group and only in that group namespace (so in the example above: only foo/bar
)...
For now we used skip_projects
as a workaround for this, but this is too problematic at this point to keep doing that.
Hi,
Just trying out this util because of the lack of group deploy keys in Gitlab.
We have a self hosted Gitlab which is exposed using HTTPS.
I am an admin in the system and I created a personal access token with 'api' privileges.
But whenever I try and use gitlabform it errors.
$ gitlabform --debug --noop Puppet
Reading config from: /Users/cliff/config.yml
URL-encoded url=https://gitlab.company.com/api/v3/version
Starting new HTTPS connection (1): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (2): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (3): gitlab.company.com
Incremented Retry for (url='/api/v3/version'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))': /api/v3/version
Starting new HTTPS connection (4): gitlab.company.com
Aborting - GitLab test request failed, details: 'HTTPSConnectionPool(host='gitlab.company.com', port=443): Max retries exceeded with url: /api/v3/version (Caused by ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')))'
I have noticed that the password variable set in the repo CI/CD variables can't be used in the config.yml, but even when I make changes to my JIRA service settings in the config.yml the jira service remains unchanged. Do you need to manually delete the settings? Or can they be deleted before being re-applied with new settings?
Generally struggling with getting these services set up correctly. I have to replace the password below in order to avoid leaving the password in the config.yml, and replace it at the gitlab-ci.yml level with: sed -i -e 's/SED_REPLACE_JIRA_PASSWORD/'"$JIRA_PASSWORD"'/g' config.yml
.
My services configuration, changing the password doesn't seem to get applied, nor does changes to the commit events from true to false.
"group/subgroup/projectname": services: mattermost: active: true webhook: https://mattermost.com/hooks/xx9bnr7jijndtgangda64w username: gitlab merge_requests_events: true merge_request_channel: "merge-requests-channel" push_events: false issues_events: false confidential_issues_events: false tag_push_events: false note_events: false confidential_note_events: false pipeline_events: false wiki_page_events: false branches_to_be_notified: "all" jira: active: true url: http://jiralo.xxx.com username: "botaccess" password: "SED_REPLACE_JIRA_PASSWORD" merge_requests_events: true commit_events: true
(Perhaps it's out of the scope of this app and should be done with some templating tool after #5 is implemented, but I would like to describe the idea here anyway.)
It would be nice to be able to manage .gitlab-ci.yml
files in an additive way.
For example we have some tests that are shared for all repos in group X, but would like to add some additional ones only in projects X/p1 and X/p2.
Other example is that we do deployments in all repos in group X the same way, except that we don't do them at all for project X/p3.
I am thinking about using a new config section, let's say 'gitlab_ci_yaml' (this name is on purpose different that the filename so that it won't be confused with files:
entry), that could accept .gitlab-ci.yml
syntax extended with some templating features, like:
I would not use either merging or replacing by default, because the default is non-obvious and expect one of the 2 last keys to be present if you redefine some element, like a job, on a lower
Example config:
group_settings:
X:
gitlab_ci_yaml:
branches: all
yaml:
variables:
THING_A: some_value
stages:
- test
- deploy
test:
stage: test
script:
- ./run_test.sh
deployment:
stage: deploy
script:
- ./deploy.sh
project_settings:
X/p1:
gitlab_ci_yaml:
branches: all
yaml:
test_something_p1_specific:
stage: test
script:
- ./run_test_p1.sh
deployment:
gitlabform_replace: true
stage: deploy
script:
- ./deploy_p1.sh
Effective .gitlab-ci.yml
in all projects in X group except p1:
variables:
THING_A: some_value
stages:
- test
- deploy
test:
stage: test
script:
- ./run_test.sh
deployment:
stage: deploy
script:
- ./deploy.sh
..and in X/p1:
variables:
THING_A: some_value
stages:
- test
- deploy
test:
stage: test
script:
- ./run_test.sh
test_something_p1_specific:
stage: test
script:
- ./run_test_p1.sh
deployment:
stage: deploy
script:
- ./deploy_p1.sh
If anyone reads this :P then please let me know what you think about this proposal.
Hello,
I've tried upgrading from v1.6.0 to 1.8.0 and encountered problems with it.
Stack trace:
*** # of groups to process: 39
*** # of projects to process: 238
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/config-Z17lvQEG/bin/gitlabform", line 6, in <module>
GitLabForm().main()
File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 135, in main
self.process_all(projects_and_groups, groups)
File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 197, in process_all
configuration = self.c.get_effective_config_for_group(group)
File "/root/.local/share/virtualenvs/config-Z17lvQEG/lib/python3.7/site-packages/gitlabform/configuration/projects_and_groups.py", line 111, in get_effective_config_for_group
group_config = self.get_config_for_group(group)
AttributeError: 'Configuration' object has no attribute 'get_config_for_group'
In our GitLab instance in some of our groups managed with gitlabform
, we have almost 100 projects, sometimes with more than 100 branches.
Because of that if we change a few files by config files:
section change (.gitlab-ci.yml
and some related files) we create A LOT of new pipelines in GitLab within just seconds during gitlabform
run.
Even with automatic pipeline cancellation this has caused our GitLab instance to become unresponsive and start returning 503 for subsequent API calls.
I have thought about making changes in files within a single branch with a single commit but as we would have to switch from using https://docs.gitlab.com/ee/api/repository_files.html#update-existing-file-in-repository to commandline git, it would be quite hard.
Also the benefits would not be big now that we have auto cancelling redundant pipelines in GitLab, since 9.1 IIRC.
Therefore as a workaround for this issue I would wrap changing files stage with disabling pipelines and re-enabling them afterwards.
(I thought about some way to disable writes to this repo in the mean time to minimize possibility that someone will push their commits into the repo while pipelines are disabled, but I can't use archiving feature because that would prevent gitlabform
from editing the files too and I don't want to start messing with project membership at this point.)
See #30 for rationale.
Gitlab has some settings for a group themselves. However, when I try to set up some hook with gitlab-conf, settings are applied for project under the group instead. To be specific, a webhook can be installed at group level, or project level; however specifying hook under groups_settings
will deploy the hook for every project under the group.
Relevant source: https://github.com/egnyte/gitlabform/blob/73acba82990655247c8fc5d5be84864ccf792a50/gitlabform/gitlabform/core.py#L210 : there isn't setting for group themselves.
Is it possible to remove keys from project/group/common?
Is it possible to terminate the program on the first error? That would be very useful when using the tool as a CI tool for gitlab per-se.
Current behavior: The program continues event if some error occurs (https://github.com/egnyte/gitlabform/blob/84b7986aacb8417021ed8ea7ee2efcc3e517506b/gitlabform/gitlabform/core.py#L222)
Expected behavior: The program terminates immediately when the first error occurs.
When adjusting the Jira service the commit_events option is always true, and has to be set to false manually through the UI, the same is true for the mattermost service and the confidential_issues_events option.
project_settings:
"group/subgroup/project":
services:
mattermost:
active: true
webhook: https://mattermost.com/hooks/xxx
username: gitlab
merge_requests_events: true
merge_request_channel: "merge-requests"
push_events: false
issues_events: false
confidential_issues_events: false # This doesnt appear to work, is always true, in all cases
tag_push_events: false
note_events: false
confidential_note_events: false
pipeline_events: false
wiki_page_events: false
branches_to_be_notified: "all"
jira:
active: true
url: http://jiralo.exfo.com
username: "user1"
password: "password"
merge_requests_events: true
commit_events: false # This doesnt appear to work, is always true, in all cases
This is more of a feature request. I came across this project while looking for tools that I could use to automatically manage the CI / CD schedules within a GitLab instance. AFAICS gitlabform
currently does not support this, but IMHO it would fit in very well.
The API is documented at: https://docs.gitlab.com/ee/api/pipeline_schedules.html
Because of the "raw parameters passing" design concept applied in this app you cannot detect all kinds of syntax errors with a reasonable amount of development effort until you try to apply the config.
But we should be able to detect some key syntax problems to prevent users from having doubts like mentioned here:
I could just be using it wrong (although I've tried moving the key "approvers" under "approvals" and this didn't work either).
Let's fix this! Ideas on how to do it are very welcome. :) I am thinking about generating the effective config and then checking it against some YAML-schema thing if something like that exists.
Can you retag as v1.0.4? Thanks!
I feel that if I document how this app is designed and provide a real-case example of how to add features to it, it will get much more PRs and with community help will improve much faster than I can do it on my own.
Lack of tests in this app is something I am ashamed of for a long time now. I used to excuse myself by stating that running Gitlabform at Egnyte for >800 repositories every day in our CI is some way of testing it.
And partially it is but it does not detect all issues.
Some issues are related to things that used to work stop working and we do not notice that because we did not reconfigure repos to a different config to notice lack of change. This is is the case of not noticing issues like #68.
Other issues are problems related to apparent GitLab API regressions. This is the case of #69.
Therefore I would like to add integration acceptance tests that will:
Apart from running them after each commit I want to set up a schedul to run it every day or every week to detect problems caused by changes in the new GitLab versions.
UPDATE: I was pointed out that these are not integrations tests but rather acceptance tests. Thanks for that!
I'm receiving the following error when trying to apply against a newly created repo.
โ gitlab-config git:(master!) gitlabform -c config.yml ALL_DEFINED
>>> Processing ALL groups and projects defined in config
*** # of groups to process: 50
*** # of projects to process: 18
* [1/18] Processing: company/technology/dev/foobar/foobar-ab-library
* [2/18] Processing: company/technology/dev/foobar/foobar-admin-gateway
* [3/18] Processing: company/technology/dev/foobar/foobar-enrollment
* [4/18] Processing: company/technology/dev/foobar/foobar-fi-config
* [5/18] Processing: company/technology/dev/foobar/foobar-fi-config-web
* [6/18] Processing: company/technology/dev/foobar/foobar-gateway
* [7/18] Processing: company/technology/dev/foobar/foobar-mobile-app
* [8/18] Processing: company/technology/dev/foobar/foobar-profile
* [9/18] Processing: company/technology/dev/foobar/pfm_demo
* [10/18] Processing: company/technology/dev/pi/test
Traceback (most recent call last):
File "/usr/local/bin/gitlabform", line 6, in <module>
GitLabForm().main()
File "/usr/local/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 136, in main
self.process_all(projects_and_groups)
File "/usr/local/lib/python3.7/site-packages/gitlabform/gitlabform/core.py", line 201, in process_all
configuration = self.c.get_effective_config_for_project(project_and_group)
File "/usr/local/lib/python3.7/site-packages/gitlabform/configuration/projects_and_groups.py", line 54, in get_effective_config_for_project
for key in group_config.keys() | common_config.keys():
AttributeError: 'NoneType' object has no attribute 'keys'
Deleting the repo or commenting out that group in the config allows for a successful run.
CONFIG
# GitLab API access config
gitlab:
url: https://gitlab.com
token: ""
api_version: 4
#
# You can define settings on 3 levels here:
# * common - for ALL projects in ALL groups
# * group - for ALL projects in selected groups
# * project - for single projects
#
# Exception: when `skip: true` is set on a more specific level, given config section is not set AT ALL for given project.
#
common_settings:
project_settings:
jobs_enabled: true
public_jobs: false
visibility: private
default_branch: master
issues_enabled: false
snippets_enabled: false
wiki_enabled: false
shared_runners_enabled: false
only_allow_merge_if_pipeline_succeeds: true
only_allow_merge_if_all_discussions_are_resolved: false
merge_requests:
approvals:
approvals_before_merge: 2
reset_approvals_on_push: false
disable_overriding_approvers_per_merge_request: true
project_push_rules:
prevent_secrets: true
author_email_regex: ""
branches:
master:
protected: true
developers_can_push: false
developers_can_merge: true
group_settings:
############################################
# Developer Related Repos
############################################
"company/technology/dev/an":
"company/technology/dev/apt":
"company/technology/dev/auto":
"company/technology/dev/bcm":
"company/technology/dev/cdn":
"company/technology/dev/chat":
"company/technology/dev/cms":
"company/technology/dev/com":
"company/technology/dev/crm":
"company/technology/dev/dep":
"company/technology/dev/dq":
"company/technology/dev/dw":
"company/technology/dev/edw":
"company/technology/dev/fed":
"company/technology/dev/fid":
"company/technology/dev/gen2":
"company/technology/dev/gov":
"company/technology/dev/hack":
"company/technology/dev/hub":
"company/technology/dev/kcas":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/dev/kloan":
"company/technology/dev/lamp":
"company/technology/dev/lemp":
"company/technology/dev/llib":
"company/technology/dev/map":
"company/technology/dev/ms":
"company/technology/dev/op":
"company/technology/dev/pam":
"company/technology/dev/pdm":
"company/technology/dev/phpv":
"company/technology/dev/pi":
"company/technology/dev/pmt":
"company/technology/dev/poc":
"company/technology/dev/ppm":
"company/technology/dev/pres":
"company/technology/dev/rct":
"company/technology/dev/rcx":
"company/technology/dev/res":
"company/technology/dev/seed":
"company/technology/dev/stat":
"company/technology/dev/sui":
"company/technology/dev/svc":
"company/technology/dev/tran":
############################################
# Infrastructure Related Repos
############################################
# Packer Projects
"company/technology/infra/ami":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/ans":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/aws":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/cfg":
"company/technology/infra/chef":
"company/technology/infra/cloud-ops":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/doc":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/dpk":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/glci":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/helm":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/pup":
"company/technology/infra/run":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/sys":
"company/technology/infra/ter":
services:
slack:
webhook: ""
username: "Gitlab Bot"
channel: ""
notify_only_default_branch: true
confidential_issues_events: false
issues_events: false
merge_requests_events: true
note_events: false
pipeline_events: true
push_events: false
tag_push_events: false
wiki_page_events: false
"company/technology/infra/tut":
Hi guys,
Would it be possible to add the following resource to the tool? POST /projects/:id/approvals
Potential problem: It is only available to EE starter
and above. Not sure how you handle this in the tool...
Let me know if I can be of any help! I am not very familiar with Python, but I'm sure I could make something work.
Cheers!
Currently you don't seem to be able to operate against groups with spaces.
Setting branches
! Branch 'feature/some_branch_name' not found when trying to set it as protected/unprotected
For further investigation:
* [215/420] Processing: internal_apps/watchdog
+++ Error while processing 'internal_apps/watchdog'
Traceback (most recent call last):
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 211, in process_all
self.process_deploy_keys(project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 31, in method_wrapper
return method(self, project_and_group, configuration)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlabform/core.py", line 235, in process_deploy_keys
self.gl.post_deploy_key(project_and_group, configuration['deploy_keys'][deploy_key])
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/projects.py", line 40, in post_deploy_key
self._make_requests_to_api("projects/%s/deploy_keys", pid, 'POST', deploy_key, expected_codes=201)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 47, in _make_requests_to_api
response = self._make_request_to_api(path_as_format_string, args, method, data, expected_codes)
File "/srv/gitlab_tools/venv/lib64/python3.5/site-packages/gitlabform/gitlab/core.py", line 83, in _make_request_to_api
raise e
gitlabform.gitlab.core.UnexpectedResponseException: Request url='https://xxx/api/v3/projects/1144/deploy_keys', method=POST, data='{'key': 'ssh-ed25519 yyyyyyyyyyy [email protected]', 'can_push': True, 'title': '[email protected] auto-bump key'}' failed - expected code(s) 201, got code 400 & body: 'b'{"message":{"fingerprint":["has already been taken"]}}''
Does it make sense to add the ability to create branches in the tool?
I ask because we want to use it to keep our repo settings in line as people create repos in our namesapce, therefore the ideal flow would be:
repo created > hook fired > gitlabform run > master branch is created and branch protection enabled.
When I started working on Gitlabform (as an internal Egnyte app back then) in the beginning of 2017, one existing GitLab python library I have found had a README that stated something like "Screw it, I am not maintaining this library any longer - GitLab makes too many API changes between even patch versions to keep up with it!"... ๐
I also remember that some other library I considered was hiding too many GitLab API internals so it was not compatible with the design decision I made to forward values from the YAML config verbatim as parameters for PUTs to GitLab API endpoints (to not have to update Gitlabform app in case of those dreaded API changes that I was warned about above).
Therefore I have started to just use requests
and wrote my own library.
I don't remember if I have even found the python-gitlab/python-gitlab and why I have not considered using it...
But now as it has gained friction (1.1k stars as of now) it would be good to stop reinventing the wheel and switch to it. If it lacks something that I need, then I should write a PR to add it to it.
For now I see that is DOES provide both creating new entities by providing raw JSON of parameters as well a direct access to parameters by the same names as in the GitLab API (see examples in https://python-gitlab.readthedocs.io/en/latest/gl_objects/projects.html vs https://docs.gitlab.com/ee/api/projects.html#edit-project) so it seems to be possible relatively easily.
In files
section, for branches
key we now support array of branch names or a string "all".
It would be useful to be able to also use a string "protected" that would make the file be managed only in protected branches.
Hi,
I read the documentation up and down but could not find how to do what described here https://docs.gitlab.com/ee/api/protected_branches.html#protect-repository-branches. Is this not implemented yet? Thanx
When installing gitlabform on MacOS (High Sierra), I'm facing problems.
As it happens, the specified version (3.12) is affected by problems described in yaml/pyyaml#15
therefore gitlabform fails to install with pip as well.
The solution seems to be simple - update the dependency to pyyaml 3.13 which solves the problem, or don't requre a specific version in the first place.
All my setups are according to the documentation, yet, it fails to find the config.yml.
โ gitlab pwd
/Users/Nomadme/Projects/Code/gitlab
โ gitlab ll
total 8
drwxr-xr-x 3 Nomadme staff 102 Jan 29 10:15 ./
drwxr-xr-x 27 Nomadme staff 918 Jan 29 09:49 ../
-rw-r--r-- 1 Nomadme staff 974 Jan 29 10:14 config.yml
โ gitlab gitlabform ALL
Aborting - config file not found at: /Users/Nomadme/Projects/Code/gitlab/config.yml
Applying a configuration using approvers
and approver_groups
has no effect while running against Gitlab 12.3.5-ee.
The config I'm using:
group_settings:
'very-silly-test-group-for-ldap-and-fun':
merge_requests:
approvals:
approvals_before_merge: 1
reset_approvals_on_push: false
disable_overriding_approvers_per_merge_request: false
merge_requests_author_approval: true
approver_groups:
- gitlab
approvers:
- waldekm
- foouser
With the log
$ gitlabform -vvv -c config.yml very-silly-test-group-for-ldap-and-fun
Reading config from: /Users/waldekm/git/config/config.yml
Connected to GitLab version: 12.3.5-ee (9dbaa740018)
Config file is declared to be compatible with GitLab API v4
Reading config from: /Users/waldekm/git/config/config.yml
*** # of groups to process: 1
*** # of projects to process: 1
* [1/1] Processing: very-silly-test-group-for-ldap-and-fun/testy-project
Setting project_settings
Setting project settings: {'only_allow_merge_if_pipeline_succeeds': True, 'only_allow_merge_if_all_discussions_are_resolved': True, 'shared_runners_enabled': True, 'request_access_enabled': True, 'wiki_enabled': False, 'snippets_enabled': False}
Setting merge_requests
Setting approvals settings: {'approvals_before_merge': 1, 'reset_approvals_on_push': False, 'disable_overriding_approvers_per_merge_request': False, 'merge_requests_author_approval': True}
Setting approvers to users ['waldekm', 'foouser'] and groups ['gitlab']
Setting hooks
Gitlabform implementation uses endpoint projects/:id/approvers
which has been deprecated:
https://docs.gitlab.com/ee/api/merge_request_approvals.html#change-allowed-approvers
I could just be using it wrong (although I've tried moving the key approvers
under approvals
and this didn't work either). It might be also that the deprecated endpoint doesn't work anymore. In any case, adding/removing users to the approvers
list and running gitlabform doesn't show in gitlab-rails/api_json.log
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.