gotsunami / gitlab-copy Goto Github PK

View Code? Open in Web Editor NEW

126.0 7.0 20.0 603 KB

Batch copy issues between GitLab instances, with some cool features

License: MIT License

Go 92.30% Makefile 4.68% Shell 3.02%

gitlab batch-copy cli

gitlab-copy's People

Contributors

Stargazers

Watchers

Forkers

egmanoj zhongsihang namawakame rowhit cacciaresi houssoli falktannhaeuser liyingzh skwi pks-os cesaremorel haghighi suzanneembury corentinrx grantx2016 zubastic ptzagk matt-theforge fingthinking

gitlab-copy's Issues

GitLab.com rate limits kick in

According to https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits, there is now a 60/min rate limit on note creation.

When using this tool, I get errors like:

2021/11/09 11:28:46 target: error creating note for issue #7: POST https://gitlab.com/api/v4/projects/.../issues/7/notes: 429 {message: {error: This endpoint has been requested too many times. Try again later.}}

This is unfortunate because the next time the tool is started it will start from the next issue, so there may be notes missing from the previous issue.

Maybe a 1s delay can be added when adding notes?

Add Go report card badge into README

Add Go report card badge
Fix version numbers in README and ChangeLog

migrate: panic on nil pointer

Hi @matm,

With the latest update, 0.8.1, I have run the gitlab-copy to copy from one Gitlab instance to another one.

First issue is copied successfully. But when it started copying second issue, I got this error:

...
Copying issues ...
2023/03/10 06:03:28 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=1&per_page=100&sort=asc
2023/03/10 06:03:30 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=2&per_page=100&sort=asc
2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=3&per_page=100&sort=asc
2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues/1
2023/03/10 06:03:31 [DEBUG] GET https://mylocalserver.com/api/v4/projects/68/issues
target: issue 1 already exists, skipping...2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues/2
2023/03/10 06:03:32 [DEBUG] GET https://mylocalserver.com/api/v4/projects/68/issues
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x50 pc=0xcc334d]

goroutine 1 [running]:
github.com/gotsunami/gitlab-copy/migration.(*Migration).migrateIssue(0xc00008b110, 0xc0000c8008?)
        /home/mathias/progs/gitlab-copy/migration/issue.go:135 +0x32d
github.com/gotsunami/gitlab-copy/migration.(*Migration).Migrate(0xc00008b110)
        /home/mathias/progs/gitlab-copy/migration/issue.go:406 +0x778
main.main()
        /home/mathias/progs/gitlab-copy/cmd/gitlab-copy/main.go:172 +0x1065

Copy labels from globals

I want to define labels at the site level and just copy to each project. How can I copy labels without specifying a project?

Add a reference in the old issue pointing to the new one

and an option to close the old one.

Preserve order of issues

Can the order of issues be preserved? This will help preserve issue numbers when copying issues from an existing project to a project that doesn't have any.

Copy issues within the same project (aka cloning)

I am looking for a method to copy my issues within the same project:

from:
  url: https://gitlab.com
  token: my-very-secret-token
  project: my-team/my-project
  issues:
  - 1156
to:
  url: https://gitlab.com
  token: my-very-secret-token
  project: my-team/my-project

unfortunately this gives me an error: target: issue 1156 already exists, skipping...

Is this feature possible and do you have time to add this?

Not all labels are copied

Thanks for the wonderful tool!

I tried a YAML setup like this to copy labels between projects:

from:
  url: http://gitlab.foobar.com
  token: foo
  project: 1
  labelsOnly: true
to:
  url: http://gitlab.foobar.com
  token: foo
  project: 2

Unfortunately, only the first 20 labels were copied, due to pagination.

An partial quick-fix might be to use per_page=100 to at least increase the number of labels copied, and update "... in order to copy all labels from one project to another ..." at https://github.com/gotsunami/gitlab-copy#more-features.

See:
https://docs.gitlab.com/ee/api/labels.html#list-labels
https://docs.gitlab.com/ee/api/index.html#offset-based-pagination

I'm sorry, I don't speak Go at all so I can't offer a pull request. Thanks for all your hard work!

User ID Mapping

I can understand why you added the "User Name @userid Wrote..." on each comment, that makes sense to not have to workout the actual db ids of the users. However, I wonder if the YML could include some type of user id mapping, so that you don't have as much work to do to workout the mapping in the code.

Project is dead?

Hello @cacciaresi @egmanoj @matm @jflemer-ndp @aboutofpluto

It looks like last update was 4 years ago.

Is there a plan to update the project or is it completely abandoned?

Thanks

Current version overwrites From host information with To host information

Very happy to have found this utility. It helped me to solve capturing a snapshot of a repo that was too large to export/import by bringing over just the current state and then using this tool to get the issues brought over.

After successfully testing gitlab-copy and getting it to work on sample repos that both lived under the same base host (and shared a token) I wasn't initially successful when I tried using https://gitlab.host.a to https://gitlab.host.b. The error shown was that the project couldn't be found.

I finally figured out how to build and use a local version of the code (I am a complete 'go' novice) so I could add print statements to see why and found that the Migration had the DstClient token and baseURL stored in the SrcClient.
It seeems the new refactoring of the GitLaber in to gitlab/client.go causes the fromgl to get overwritten by the togl values in migration/issues.go when creating. a new migration.
In gitlab/client.go the init() sets the DefaultClient

DefaultClient = new(Client).

In migrations/issue.go:

fromgl := gitlab.DefaultClientFrom.New(nil, c.SrcPrj.Token)
...
togl := gitlab.DefaultClientTo.New(nil, c.DstPrj.Token)

The setting of togl changes the ptr-value of DefaultClient & fromgl as well

Again, I'm new to Go so I may be missing something obvious, however here's what the print output shows. (names modified to be generic and tokens redacted)
show-failure-config.yml:

from:
    url: https://git.from.co
    project: group_name/myproject
    token: from-token-string
to:
    url: https://gitlab.to.com
    token: to-token-string
    project: group_to/subgroup/otherproject

And the output:

~/go/src/github.com/gotsunami/gitlab-copy/bin/gitlab-copy show-failure-config.yml
DUMMY MODE: won't apply anything (stats only)
--
Creating a new Migration...
DefaultClient address: &{<nil>}
setting fromgl from 'gitlab.DefaultCient.New'
fromgl address: &{0xc4200d01e0}
DefaultClient address: &{0xc4200d01e0}
setting togl from 'gitlab.DefaultCient.New'
togl address: &{0xc4200d03c0}
DefaultClient address: &{0xc4200d03c0}
fromgl address: &{0xc4200d03c0}
====
fromgl.BaseURL is: https://gitlab.to.com/api/v4/
togl.BaseURL is: https://gitlab.to.com/api/v4/

2021/07/29 19:41:23 source project 'group_name/myproject' not found

For my immediate needs I worked around the issue by creating two additional DefaultClient vars: DefaultClientTo and DefaultClientFrom and initializing them in init() and using them to set the fromgl and togl to become the Src and Dst in EndPoint. However I think the right fix is more in line with creating a new GitLaber client for each item. I wasn't sure why the package var DefaultClient was involved.

As a final note, even after I got it working with my hack, I wasn't able to copy attachments to notes. I saw this had been reported as fixed with #15 but it wasn't working for me. I figured I'd be happy with what I got.

Option to prevent users from getting spammed by notifications

Currently if an issue is copied all notes are copied with it including developers assigning the original issue to a milestone. Is it possible to include an option to not have these notes be created as comments on destination issue so as to prevent users from being spammed with 60+ todos/emails/notifications when multiple issues they worked on in the past have been moved to a new project. Chances are they are not working on that project anymore either and don't need to be re-notified.

This image below shows what the comments look like after being copied over. Having the issue simply read "@z changed milestone to Open Source Intelligence Gathering 4 hours ago" without the original developer tag or what the user "wrote".

"error closing issue" with gitlab API v4

Thank alot for this tool, it saved me a bunch of work !

However for gitlab API v4, it seems to have problems on closing bugs.
I just closed them by hand now, which was no big deal for only ~20 bugs. Here the console output:

schwinn@belpc077:~/git/src/github.com/gotsunami/gitlab-copy$ ./bin/gitlab-copy -y gitlab.yml 
source: aschwinn/silecs at https://gitlab.cern.ch/api/v4
target: al.schwinn/silecs at https://gitlab.com/api/v4
--
source: finding issues ... OK
source: 46 issues (26 opened, 20 closed)
source: 3 milestone(s): 2.1.0, 3.0.0, 2.1.2
source: 7 label(s): virtual-controller, mergeWithCERN, silecs-cli-client, silecs-codegen, silecs-communication-cpp, silecs-diagnostic-cpp, silecs-model
source: 237 notes                                                  
--
Copying labels ...
Copying issues ...
2018/09/04 09:45:30 target: error closing issue #1: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882209: 404 {message: 404 Not found}
2018/09/04 09:45:38 target: error closing issue #2: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882210: 404 {message: 404 Not found}
2018/09/04 09:45:44 target: error closing issue #3: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882212: 404 {message: 404 Not found}
target: created issue #4: Build unit-tests to compile ( and link ? ) a DU-wrapper [opened]
target: created issue #5: Provide a way to have lab-specific code-generation [opened]
2018/09/04 09:45:59 target: error closing issue #6: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882218: 404 {message: 404 Not found}
2018/09/04 09:46:05 target: error closing issue #7: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882222: 404 {message: 404 Not found}
2018/09/04 09:46:09 target: error closing issue #8: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882225: 404 {message: 404 Not found}
target: created issue #9: Add support for boolean data-type [opened]
2018/09/04 09:46:15 target: error closing issue #10: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882228: 404 {message: 404 Not found}
target: created issue #11: show in type-selection which FESA-type corresponds to which PLC-type [opened]
target: created issue #12: Use xsd naming for C++ "access types" [opened]
target: created issue #13: Restructure C++ Classes in order to reflect SilecsClasses [opened]
target: created issue #14: replace all tabs by spaces [opened]
target: created issue #15: Usage of c-templates for different register-data types [opened]
target: created issue #16: Support for enums missing [opened]
target: created issue #17: restructure class "SilecsCluster" and rename it to "SilecsDesign" [opened]
target: created issue #18: Allow to give alias-names for fesa-classes and DU's [opened]
target: created issue #19: Provide template-builder which can re-use arguments [opened]
target: created issue #20: Introduce package "virtual controller" [opened]
2018/09/04 09:46:43 target: error closing issue #21: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882327: 404 {message: 404 Not found}
target: created issue #22: Re-Structure C++ Block model [opened]
target: created issue #23: move generated code for fesa-class into "generated" folder [opened]
target: created issue #24: Split library into snap7 / modbus / etc [opened]
target: created issue #25: Investigate: How to make use of BLOCK_MODE in FESA [opened]
2018/09/04 09:46:55 target: error closing issue #26: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882336: 404 {message: 404 Not found}
2018/09/04 09:46:57 target: error closing issue #27: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882337: 404 {message: 404 Not found}
target: created issue #28: Provide lab-specific install-script [opened]
target: created issue #29: Provide a compartibility-mode for old checksums [opened]
target: created issue #30: Create merge requests for differences between CERN silecs-communication-cpp and GSI version [opened]
2018/09/04 09:47:12 target: error closing issue #31: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882342: 404 {message: 404 Not found}
target: created issue #32: Create merge requests for differences between CERN silecs-diagnostic-cpp and GSI version [opened]
2018/09/04 09:47:21 target: error closing issue #33: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882351: 404 {message: 404 Not found}
2018/09/04 09:47:25 target: error closing issue #34: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882354: 404 {message: 404 Not found}
2018/09/04 09:47:28 target: error closing issue #35: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882356: 404 {message: 404 Not found}
2018/09/04 09:47:32 target: error closing issue #36: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882358: 404 {message: 404 Not found}
target: created issue #37: Evaluate possibility to connect memory register with outputs in Siemens scl code [opened]
2018/09/04 09:47:36 target: error closing issue #38: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882362: 404 {message: 404 Not found}
target: created issue #39: Make use of python package 'unittest' [opened]
target: created issue #40: move  string to value method(s) to silecs-communication utils [opened]
2018/09/04 09:47:48 target: error closing issue #41: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882367: 404 {message: 404 Not found}
target: created issue #42: isInitialized_ check in codegen should get it's own method [opened]
2018/09/04 09:47:54 target: error closing issue #43: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882372: 404 {message: 404 Not found}
2018/09/04 09:47:59 target: error closing issue #44: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882374: 404 {message: 404 Not found}
2018/09/04 09:48:04 target: error closing issue #45: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882376: 404 {message: 404 Not found}
target: created issue #46: Prefix Block Makros in generated ClassHeader with Classname [opened]

Unused Labels not copied

If a label is not yet used, it is not copied over. Would be great to have it.

Issue numbering off by one

I copied a repo with 671 issues from private server of old developers to gitlab.com. The new repo has 670 issues. The copying has skipped issue 199, meaning all issues after that are off by 1, so cross-referencing of issues above 199 are wrong, which complicates matters! :-D

I can't see anything special about that issue, it just contains a couple of lines and a few comments...

Label Descriptions aren't copied

Coping labels from one project to another works good except of the label descriptions. When opening the label settings I can see the copied labels with correct colors, but they don't contain any description.

This should be fixed for using the tool in production.

Copy issues with specific label

Hi,
I think it would be usefull to have an option in the gitlab.yml file to indicate that we want to copy issues with a specific label.

Like this, we don't have to deal with issue numbers when we have new issues to copy from one project to another one, we only have to apply a tag.

Cleaner logging in the terminal, with verbosity level

Write cleaner log messages in the terminal
New -v CLI flag to increase log level to DEBUG

Add mocking mechanism and write unit tests

Let's mock part of the GitLab library used and write unit tests. Too many failures reported related to nil pointer dereference.

Linked attachments are not transferred

I've run gitlab-copy between two GitLab v8.8.3 to transfer issues from one to the other.

Unfortunately, files that were attached to the source issues are not transferred to the new server. When referring to them in an issue (for example, showing a screenshot), the link is broken.

Remove trailing slash in server URI

To prevent getting 401 errors from GitLab when defining API path.

Add a move issues capability

Right now, issues are copied from one project to another. Let's add a -move option on the CLI so that source issues are deleted after successful creation on target project.

Logging in to Gitlab

This is not an issue, but rather a request for info/documentation. I'm trying to copy issues from one repository to another in the same gitlab instance. The instance needs users to log in first. Is there a way to accomplish this?

panic: runtime error: invalid memory address during copy

Compiled master from source to have support for API v4
Testrun was fine, but with -y I got the following runtime error:

Any help would be welcome !

schwinn@belpc077:~/git/src/github.com/gotsunami/gitlab-copy/bin$ ./gitlab-copy -y gitlab.yml
source: aschwinn/silecs at https://gitlab.cern.ch/api/v4
target: al.schwinn/silecs at https://gitlab.com/api/v4
--
source: finding issues ... OK
source: 46 issues (26 opened, 20 closed)
source: 3 milestone(s): 3.0.0, 2.1.2, 2.1.0
source: 7 label(s): mergeWithCERN, silecs-cli-client, silecs-codegen, silecs-communication-cpp, silecs-diagnostic-cpp, silecs-model, virtual-controller
source: 237 notes                                                  
--
Copying labels ...
Copying issues ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x729b45]

goroutine 1 [running]:
main.(*migration).migrateIssue(0xc4200892f0, 0x1, 0x866901, 0xc42022d560)
	/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:138 +0x3c5
main.(*migration).migrate(0xc4200892f0, 0x1, 0x1)
	/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:375 +0xf22
main.main()
	/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/main.go:156 +0x933

gitlab-copy crashes if an issue is not assigned to a milestone

Well it might not be considered as good style to not assign an issue to a milestone, but gitlab-copy should not crash with a nil pointer dereference when copying an issue that is not assigned to a milestone.

Workaround: assing all issues to a milestone

Otherwise, nice tool - keep on going! :)

Self signed SSL Support

Current project doesn't supports Gitlab instances which has self-signed SSL certificates.

That would be great if you can add support. Thank you.

GitLab API sometimes returns HTTP error 414

This occurs when an issue's description or a note's body is too large. This makes the issue creation on target fail. This error can be caught and fixed by saving a shortened description or body content.

Multiple Instance Issue

I am trying to run gitlab-copy according to this config:

from:
  url: https://mylocalgitlab.com
  token: glpat-sdfdsfsdfsdfsdfsdf
  project: namespace/project
to:
  url: https://anotherinstance.com
  token: glpat-sdfsdfsdffsdfsdfsfd
  project: namespace/project

If both instance (from and to) is same instance it works.

If instances are different, like in the example, it gave source project 'namespace/project' not found error.

Is gitlab-copy working across different instances?

HTTPS Insecure option required

If the gitlab instance uses a company CA-signed certificate (which are often not then signed by a global CA) then certificate cannot be validated and the process fails

Can you add a option type to the config.yml to skip verification? Kind of equivalent to wget's no-check-certificate flag

Output when such a case arises looks like:

DUMMY MODE: won't apply anything (stats only)
--
2018/05/30 21:19:38 network error: Get https:/api/v3/projects/namespace%2Fproject: x509: certificate signed by unknown authority

Use dep to vendor dependencies

Let's use dep for now.

Windows executable file should be renamed from gitlab-copy to gitlab-copy.exe

Labels are not applied to closed issues

When migrating issues from one project to another labels are not applied to issues that are already closed. Open issues are correctly labelled.

Fix dependencies security issues

YAML Go package vulnerable to denial of service
Excessive Platform Resource Consumption within a Loop in Kubernetes (related to YAML package too)
Uncontrolled Resource Consumption (in golang.org/x/net)

Actions

Upgrade gopkg.in/yaml.v3 v2.1.0 => v3.0.1
Upgrade golang.org/x/net v0.4.0 => v0.7.0

raise error when I migrate issue in v0.7-rc2

error such as below raised when I migrate issue by v0.7-rc2. this error occured on windows and linux.

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
main.(*migration).migrateIssue
        /home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:138 main.(*migration).migrate
        /home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:375 main.main
        /home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/main.go:156

Update go-gitlab library to v0.5.1

Update this dependency before working on #18.

Linked attachments on the issues/notes aren't transferred

Hello @matm,

When I run gitlab-copy to transfer between different instances/hosts of Gitlab, linked attachments on the issues or notes aren't transferred.

Also, this comment points out the same problem: #15 (comment)

Sign in Error while execute the command

Hi,

i very happy to found this. I wrote the gitlab.yml and executed the command but i get the following error GET https://gitlab.com/soemthing/api/v4/projects/xx/some-project: 401 {error: You need to sign in or sign up before continuing.}

I am logged in and the token is freshly made with all permissions.

i hope you can help me here.

edit: my adress in the yaml file is https://gitlab.com/mydomain and not https://gitlab.mydomain.com is that a problem?

Kind Regards
Julian

Copy uploads

Uploads are not carried across. A lot of gitlab issues contain screenshots, so it would help if these were copied over too.

build: refac the way git revisions are passed to the compiler

Remove tools/version.sh and use a different way of passing Version, GitRevision, GitBranch and Built
Update the Makefile's dist target to generate checksum files

This will make the automatic building of the tool pass on GitHub actions.

migrate milestonesOnly for only opened milestones

Relates to #13

Is there a way that a flag could be added to migrate only open milestones? Right not it copies all, but it would be nice to be able to migrate just open ones.

Thank you!

0 Issues found from Source. (Different Instance)

I tried to move all issues in one repo. from my gitlab instance to Gitlab Main. But I just got message like:

source: finding issues ... OK
source: 0 issues (0 opened, 0 closed)
source: 0 notes

And if I put source and target same my instance url then it works

I think this issue related to user mapping. I can't understand If I just put target in all users tokens then how will library identify.

milestonesOnly

Hi,

great tool.
I would like to see the feature to copy milestones only - just as the labelsOnly.

Is there a chance, that this feature gets implemented within 1 or 2 weeks?

Kind regards, Karl

Use go.mod instead of Gopkg and upgrade dependencies

Uses API v3 which is now deprecated

Shouldn't it be using V4?

Emails are sent out while mirroring

When using the gitlab-copy script, for each new issue emails are sent to the people that are referenced in that issue. Preferably this should be disabled during import time (when possible).

no Go files in /root/go/src/github.com/gotsunami/gitlab-copy

I have got a trouble during install, this is the result:

#go get -u github.com/gotsunami/gitlab-copy

´´´package github.com/gotsunami/gitlab-copy: no Go files in /root/go/src/github.com/gotsunami/gitlab-copy
´´´

Add support for auto update

Check whether a new version of the tool is available. With a periodic check and a dedicated update command.

Label Colors

When doing the copy, the label color is created on target with default color, and doesn't retain its original color from source.

On a side note, great tool, saved us a lot of time. Thank you.

Copy labels only between projects

Add a labelsOnly entry in config file to easily copy labels between two projects:

from:
  url: https://gitlab.mydomain.com
  token: atoken
  project: namespace/project
  labelsOnly: true
to:
  url: https://gitlab.sameorotherdomain.com
  token: anothertoken
  project: namespace/project

Label attributes are conserved.

Copy Issues Without Notes/Comments

Hi there,

Is there a way to copy Notes/Comments from Gitlab Issues WITHOUT copying over all of the notes and comments? I have had a look through the project and I can't see anyway to do this other than editing the source code.

Thanks!