rvojcik/gitlab-project-export

Simple python project for exporting GitLab projects with Export Project funkcionality in GitLab API. Import of exported projects is also supported

gitlab-project-export's Issues

Abilty to import all projects withought having the list them

is there a way to have it import all projects from all users and groups. only way i figured out was listing out each user then using * behind it but just doing * or / both error out

Abilty to import all projects withought having the list them

Throttle requests

Trying to download many projects results in

Export failed for project <group>/<project>
API not respond well with 429 {"message":{"error":"This endpoint has been requested too many times. Try again later."}}

Add an option to specify the max number of tries for project export

Currently, code is able to download small projects.
But for large projects, it needs to wait for more.
Add an option for the user to specify the max number of tries in the YAML file and use it in project_export method.

I have an error when exporting

python --version fish-1 | 1
Python 3.6.9

/gitlab-project-export.py -c ./config.yml fish-1 | 0 [19:33:18]
Traceback (most recent call last):
File "./gitlab-project-export.py", line 49, in
c = config.Config(args.config)
File "/root/test/gitlab-project-export/gitlab_export/config.py", line 15, in init
self.config_load()
File "/root/test/gitlab-project-export/gitlab_export/config.py", line 43, in config_load
self.config = yaml.load(self.conf_fh.read(), Loader=yaml.FullLoader)
AttributeError: module 'yaml' has no attribute 'FullLoader'

error when importing

Hi, I am trying to import a repo and I am getting :

API not respond well with 404 {"message":"404 Namespace Not Found"}
{"message":"404 Namespace Not Found"}
Import was not successful

this is the command I am running:

python3 gitlab_project_import/gitlab-project-import.py -c gitlab_project_import/config_import2.yml -f ./importedgitlab_cicdtar.gz "-p group/gitlab_cicd"

I did not find anywhere where I can specify a namespace. I used the same kind of config.yml as to export the repo. Do you have an example of a specific one for importing?

Thanks Javier

Migration of Archived Projects

I've just exported from one Gitlab installation and tried to import into the new Gitlab Installation - so far this worked pretty well!

But one single project did not appear on the new Gitlab - but it was exported + imported successful without any warning. It turned out, that in the source Gitlab-Instance, this project was marked as archived.

As an improvement, it would be cool, if the exporter script either

• warns on archived projects
• ignores archived projects

Or alternatively, that the importer script would properly handle this (and the imported project would again be marked archived in the target installation, rather than getting lost after importing)

Import GitLab project error

Hi,

I am trying to export one of the gitlab project and import it.
While importing at first I got the issue,

AttributeError: module 'urllib' has no attribute 'quote'

But I resolved by changing the
urllib.quote(project_path, safe='') -> urllib.parse.quote(project_path, safe='')

Later while importing the file I am getting following issue:

File "gitlab-project-import.py", line 60, in <module>
    status = gitlab.project_import(args.project_path, args.filepath)
  File "....\gitlab-project-export\lib\gitlab.py", line 191, in project_import
    r = self.__api_import(project_name, namespace, filepath)
  File "....\gitlab-project-export\lib\gitlab.py", line 43, in __api_import
    headers=self.headers)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 516, in request
    prep = self.prepare_request(req)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 459, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 317, in prepare
    self.prepare_body(data, files, json)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 505, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "....\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 159, in _encode_files
    fdata = fp.read()
  File "....\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 111: character maps to <undefined>

Any idea how to resolve this?

Option for exporting all projects

Hi, I had to set this value to membership=False in order to download all projects on a shared gitlab instance, with config of:

  projects: 
    - '/*'

I'm not sure if anyone knows of a better way to have this utility simply export all projects?

Add topic: github-backup

Thanks for publishing this! I think it would be better findable, if displayed on GitHub.com/topics/github-backup, please consider adding that topic to this repo. Thank you :-)

Regularity of backups

Thanks for a great program!

I wanted to be able to schedule regular backups, but some projects need to be backed up at different regularities than others. So I forked and added the concept of regularity to your code. Now each project has associated with it a list of regularities. If the regularity (passed as argument -r) matches one in the list for the project, then it is exported. You can see the details at https://github.com/dazsmith/gitlab-project-export/commits/master.

Should I submit a pull request?

API not respond well with 404

Hi,

I'm trying to use the script to export my Gitlab projects, but I get API not respond well with 404 error.

I tried to specify my project as group/project-name, but I also tried with /*. Using /* it identifies all my projects and creates the folders for them (the whole group - project names hierarchy), but it gives the same 404 error for all projects.

In Gitlab I can access the project using https://_gitlab_url/group/project-name

I also tried to export the project from Postman. Using https://_gitlab_url/api/v4/projects/2/export works, but using https://_gitlab_url/api/v4/projects/group%2Fproject-name/export returns 404.

I'm using Gitlab 13.9.2-ce.0 (Dockerized).
I enabled all scopes for the Personal Access Token.

option for self-signed certs?

As many people run gitlab on private instances, they may be using self-signed certs. Is there a technique to work with this setup?

I'm thinking of a couple different solutions:

• an option to point to a cert file
• an option to disable SSL verification
• at least an update to README saying how to use REQUESTS_CA_BUNDLE to point to cert

Publish released package

It would be nice to have a pip installable version of this available to make installation more seamless than the git variant.