viperproject/bitbucket-issue-migration

Scripts for the migration from Bitbucket to GitHub.

bitbucket-issue-migration's Issues

Pull request exceptions on private repositories

Pull requests aren't being properly handled by the script, causing the following exception:

github.GithubException.GithubException: 422 {"errors": [{"field": "head", "code": "invalid", "resource": "PullRequest"}], "message": "Validation Failed", "documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request"}

Investigation must be done. The GitHub account status for 'viperproject' is 'Team', therefore the feature is supported.

Bitbucket issue 50 does not exist

❗ Bitbucket's issue number 50 does not exist!
This breaks the creation of Github issues.

Selection of the master branch

See the FIXME at

bitbucket-issue-migration/import-forks.py

Line 142 in 5f7695e

# FIXME: this doesn't correctly pick the default branch of some repos

Migration script is not handling private repositories properly

Since private repositories require authentication in order to be accessed, currently the script is causing the following exception:

HTTPError: 403 Client Error: Forbidden for url: https://api.bitbucket.org/2.0/repositories/viperproject/...

Therefore the script should be changed to include BitBucket's authorized login and app password for reading access.

Prevent email notifications during the migration

gitignore vs hgignore

Currently, the repos just have the hgignore file and no gitignore file. A probably cumbersome option would be to make the commits such that changes to hgignore files are interpreted as changes to a gitignore file. The simplest solution would be to just change this manually with the newest commit for the master branch and people would have to deal with this themselves in other cases.

Migrate participants and/or subscribe users

I created the issue https://github.com/fpoli/viper-silicon/issues/261 but I'm not marked as subscribed to it or as a participant. I assume that means that I will not be notified if something is changed or there are new comments.

Is there a way to change that and keep the people who created the issues and wrote comments updated?

Crash when processing dangling pull requests

Some opened pull requests were created before the source repository was deleted or changed into private. As a result, pull requests become dangling. It is also the case if the source branch was deleted or similar scenarios.

As this seems to be a frequent problem, it is worth to fix that in the script instead of manually fix that all over.

Porting script is crashing while processing pull requests on private repositories

The stack trace is:

Prepare github issue #116 from bitbucket pull request...
Traceback (most recent call last):
  File "./migrate-discussions.py", line 1020, in <module>
    main()
  File "./migrate-discussions.py", line 1016, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 836, in bitbucket_to_github
    gissue_or_gpull = construct_gissue_or_gpull_from_bpull(bpull, bexport, cmap, args)
  File "./migrate-discussions.py", line 702, in construct_gissue_or_gpull_from_bpull
    comments += construct_gissue_comments(bcomments, cmap, args, bexport)
  File "./migrate-discussions.py", line 575, in construct_gissue_comments
    "body": construct_gcomment_body(bcomment, bcomments, cmap, args, bexport),
  File "./migrate-discussions.py", line 299, in construct_gcomment_body
    bcomment = bexport.get_detailed_comment(bcomment)
  File "/home/pakk/migration/bitbucket-issue-migration/src/bitbucket.py", line 98, in get_detailed_comment
    return get_request_json(shallow_comment["links"]["self"]["href"])
  File "/home/pakk/migration/bitbucket-issue-migration/src/utils.py", line 18, in get_request_json
    res.raise_for_status()
  File "/home/pakk/.local/lib/python3.5/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.bitbucket.org/2.0/repositories/viperproject/voila/pullrequests/9/comments/70146327

The script crashes when there's no issue tracker

Whenever attempting to port a repository from BitBucket which contains no issue tracker, the script runs into an exception:

Traceback (most recent call last):
  File "./migrate-discussions.py", line 1007, in <module>
    main()
  File "./migrate-discussions.py", line 1003, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 794, in bitbucket_to_github
    bissues = bexport.get_issues()
  File "/home/pakk/migration/bitbucket-issue-migration/src/bitbucket.py", line 43, in get_issues
    issues = list(get_paginated_json(self.repo_url + "/issues", self.session))
  File "/home/pakk/migration/bitbucket-issue-migration/src/bitbucket.py", line 12, in get_paginated_json
    result = get_request_json(next_url, session)
  File "/home/pakk/migration/bitbucket-issue-migration/src/utils.py", line 18, in get_request_json
    res.raise_for_status()
  File "/home/pakk/.local/lib/python3.5/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.bitbucket.org/2.0/repositories/viperproject/documentation/issues
Traceback (most recent call last):
  File "main.py", line 209, in <module>
    main()
  File "main.py", line 205, in main
    ), cwd=ROOT)
  File "main.py", line 27, in execute
    check_call(cmd, *args, shell=True, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)

Gists are created as public even for private repositories

When attachments are migrated from BitBucket to GitHub, they are created as gists if they are small enough. However, even for private repositories, the gists are public. I was able to create secret gists as follows:

diff --git a/src/github.py b/src/github.py
index c0e87f3..bc97bcb 100644
--- a/src/github.py
+++ b/src/github.py
@@ -58,7 +58,7 @@ class GithubImport:
         gist = self.get_gist_by_description(gist_data["description"])
         if gist is None:
             gist = self.github.get_user().create_gist(
-                True,
+                False,
                 gist_data["files"],
                 gist_data["description"]
             )

and it seemed to work. I wondered if it would make sense to make public gists for public repos, and secret gists for private repos, but in the end decided that they should always probably be secret. Secret doesn't mean private; you can still access them if you know the URL, they are just not discoverable. And if the repo is made private in future, you probably want the gists to be secret at that point. So it's probably OK just to change True to False there, without any check on the public/private status of the repo.

Script is crashing on more complex repositories

For a variety of reasons, the script it crashing in the following repositories:

• Deprecated / rust2viper
• Deprecated / obligations/silver

The root cause must be investigated and fixed. Those two repositories must be ported as a result.

Slow the script down

Hello,
I have been tasked with moving my orgs repos from bitbucket to github.

And your script has been a godsend!
I’ve moved most of my small repos to Github flawlessly

I seem to be hitting all sorts of rate limits with my large repos that have 600+ prs.
i cannot seem to delete the PRs, over there at bitbucket and github does not seem to want them at the rate the script wants to migrate them here

Is there any way I could ask the script to slow down?

and THANK YOU so, so, so much for making this!

Conversion of issue links

The issues mentioned in the migration of carbon issue 229 are not recognized and converted to url. So, Github renders it differently depending on the order of creation of the issues. An example of the result:

• issue #126
• issue #132
• issue #174
• issue #220

Validation Failed Github Exception

Exception while migrating Silicon:

Migrate attachments for bitbucket issue #335... [rate limiting: 4595]
Traceback (most recent call last):
  File "./migrate-discussions.py", line 883, in <module>
    main()
  File "./migrate-discussions.py", line 879, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 693, in bitbucket_to_github
    gist = gimport.get_or_create_gist_by_description(gist_data)
  File "/home/fpoli/src/bitbucket-issue-migration/src/github.py", line 54, in get_or_create_gist_by_description
    gist_data["description"]
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/AuthenticatedUser.py", line 532, in create_gist
    "POST", "/gists", input=post_parameters
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 322, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 345, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 422 {"message": "Validation Failed", "documentation_url": "https://developer.github.com/v3/gists/#create-a-gist", "errors": [{"field": "files", "resource": "Gist", "code": "missing_field"}]}

Invalid pull request creation

From the migration:

Upload github issue or pull request #469... [rate limiting: 4078]
Create github pull request #469...
Traceback (most recent call last):
  File "./migrate-discussions.py", line 890, in <module>
    main()
  File "./migrate-discussions.py", line 886, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 749, in bitbucket_to_github
    gimport.create_pull_with_comments(data)
  File "/home/fpoli/src/bitbucket-issue-migration/src/github.py", line 185, in create_pull_with_comments
    head=meta["head"],
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Repository.py", line 1307, in create_pull
    return self.__create_pull_1(*args, **kwds)
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Repository.py", line 1330, in __create_pull_1
    return self.__create_pull(title=title, body=body, base=base, head=head)
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Repository.py", line 1341, in __create_pull
    "POST", self.url + "/pulls", input=post_parameters
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 322, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 345, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 422 {"documentation_url": "https://developer.github.com/v3/pulls/#create-a-pull-request", "errors": [{"resource": "PullRequest", "field": "head", "code": "invalid"}], "message": "Validation Failed"}

Tags are missing

The released versions of Silver (74329a5), Silicon (88f1d34), Carbon (ba027f2), and ViperServer (7cc7ce5) should be tagged. This should be easy to add manually as we only had one release so far. I think we should add these tags once the migration is over.

However, other tags are also missing, e.g., in Silicon. Not sure what's the simplest way, maybe also adding them manually.

There's no HTTPS alternative to access Mercurial repositories in BitBucket

If you are unable to use SSH keys on BitBucket for Mercurial (even if they work for Git repositories in BitBicuket), this script is going to crash with a "no response from server" exception. This message actually comes from Hg, but the script cannot handle.

So I think this script should be adapted for use with HTTPS, also because the user running this script is not meant to use his/her own account, but rather viper-admin's. So that would be also another reason.

Upload to GitHub is currently using SSH keys instead of access token

When uploading all commits to GitHub, the script is using the SSH keys implicitly. There's a risk this might be registered as an user action (as an effect of running the script) instead of commits done by "viper-admin" user, by means of the access token.

It is better to use HTTPS with login and password (access token in this case). As a test, this script should be able to run even when all SSH keys are removed.

Exception while migrating Silicon

Exception while migrating Silicon:

Migrate attachments for bitbucket issue #208... [rate limiting: 3633]
Traceback (most recent call last):
  File "./migrate-discussions.py", line 872, in <module>
    main()
  File "./migrate-discussions.py", line 868, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 686, in bitbucket_to_github
    gist = gimport.get_or_create_gist_by_description(gist_data)
  File "/home/fpoli/src/bitbucket-issue-migration/src/github.py", line 54, in get_or_create_gist_by_description
    gist_data["description"]
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/AuthenticatedUser.py", line 532, in create_gist
    "POST", "/gists", input=post_parameters
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 322, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/home/fpoli/.local/lib/python3.5/site-packages/github/Requester.py", line 345, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 502 {"message": "Server Error"}

Wiki is not being ported

Currently, this script is not porting wiki.

Map users

• ask for the github user of every author
• manually map authors of commits to users

After migrating the repo Github should have a list of all the authors

Change style of authors

Linking

• links to issues
• links to commits
  • it might be useful to build an independent tool that given two repositories (mercurial and git) builds a mapping of the commits
• links to PRs
• links to issues of other repositories (e.g. given a full url)
• links to commits of other repositories

Less important:

• links to users

Migrate wiki

Migrate pull requests

TypeError: Can't convert 'int' object to str implicitly

Migrate attachments for bitbucket issue #409... [rate limiting: 3102]
Prepare github issues...
Prepare github issue #1 from bitbucket issue...
Prepare github issue #2 from bitbucket issue...
Prepare github issue #3 from bitbucket issue...
Prepare github issue #4 from bitbucket issue...
Prepare github issue #5 from bitbucket issue...
Prepare github issue #6 from bitbucket issue...
Prepare github issue #7 from bitbucket issue...
Prepare github issue #8 from bitbucket issue...
Traceback (most recent call last):
  File "./migrate-discussions.py", line 951, in <module>
    main()
  File "./migrate-discussions.py", line 947, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 767, in bitbucket_to_github
    gissue = construct_gissue_from_bissue(bissue, bexport, attachment_gist_by_issue_id, cmap, args)
  File "./migrate-discussions.py", line 611, in construct_gissue_from_bissue
    comments += construct_gissue_comments(bcomments, cmap, args)
  File "./migrate-discussions.py", line 519, in construct_gissue_comments
    "body": construct_gcomment_body(bcomment, bcomments, cmap, args),
  File "./migrate-discussions.py", line 322, in construct_gcomment_body
    sb.append("" if bcomment["content"]["raw"] is None else map_content(bcomment["content"]["raw"], cmap, args))
  File "./migrate-discussions.py", line 253, in map_content
    tmp = replace_implicit_links_to_prs(tmp, args)
  File "./migrate-discussions.py", line 109, in replace_implicit_links_to_prs
    return IMPLICIT_PR_LINK_RE.sub(replace_pr_link, body)
  File "./migrate-discussions.py", line 106, in replace_pr_link
    gpr_number = bpr_nr + issues_count
TypeError: Can't convert 'int' object to str implicitly

Create a user for the migration

Error 500 from Bitbucket

Migrate attachments for bitbucket issue #333... [rate limiting: 4713]
Migrate attachments for bitbucket issue #334... [rate limiting: 4711]
Migrate attachments for bitbucket issue #335... [rate limiting: 4711]
Migrate attachments for bitbucket issue #336... [rate limiting: 4709]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 658, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 283, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.bitbucket.org', port=443): Max retries exceeded with url: /2.0/repositories/viperproject/silicon/issues/336/attachments/unsound.vpr (Caused by ResponseError('too many 500 error responses',))

Wrong error

The error "Error: the number of Github issues and pull requests seems to be wrong." seems to be wrong.

[2020-02-07 16:53:16] === Migrate isues and pull requests of bitbucket repository 'viperproject/viperserver' to github...
Load mapping of mercurial commits to git...
checking uniqueness of hg and git hashes...
check done
Get all bitbucket issues...
Get all 11 detailed bitbucket pull requests...
10/11...
Migrate bitbucket attachments to github...
Prepare github issues...
Prepare github issue #1 from bitbucket pull request...
Prepare github issue #2 from bitbucket pull request...
Prepare github issue #3 from bitbucket pull request...
Prepare github issue #4 from bitbucket pull request...
Prepare github issue #5 from bitbucket pull request...
Prepare github issue #6 from bitbucket pull request...
Prepare github issue #7 from bitbucket pull request...
Prepare github issue #8 from bitbucket pull request...
Prepare github issue #9 from bitbucket pull request...
Prepare github issue #10 from bitbucket pull request...
Prepare github issue #11 from bitbucket pull request...
Upload github issues...
Upload github issue or pull request #1... [rate limiting: 2925]
Create github issue #1...
Waiting...
Upload github issue or pull request #2... [rate limiting: 2925]
Create github issue #2...
Waiting...
Upload github issue or pull request #3... [rate limiting: 2925]
Create github issue #3...
Waiting...
Upload github issue or pull request #4... [rate limiting: 2925]
Create github issue #4...
Waiting...
Upload github issue or pull request #5... [rate limiting: 2925]
Create github issue #5...
Waiting...
Upload github issue or pull request #6... [rate limiting: 2925]
Create github issue #6...
Waiting...
Waiting...
Upload github issue or pull request #7... [rate limiting: 2925]
Create github issue #7...
Waiting...
Waiting...
Upload github issue or pull request #8... [rate limiting: 2925]
Create github issue #8...
Waiting...
Waiting...
Waiting...
Upload github issue or pull request #9... [rate limiting: 2925]
Create github pull request #9...
Upload github issue or pull request #10... [rate limiting: 2902]
Create github pull request #10...
Upload github issue or pull request #11... [rate limiting: 2899]
Create github pull request #11...
Error: the number of Github issues and pull requests seems to be wrong.

TypeError: int() got a None

[2020-02-17 15:26:04] === Migrate isues and pull requests of bitbucket repository 'viperproject/viperserver' to github...
Load mapping of mercurial commits to git...
checking uniqueness of hg and git hashes...
check done
Get all bitbucket issues...
Get all 11 detailed bitbucket pull requests...
10/11...
Migrate bitbucket attachments to github...
Prepare github issues...
Prepare github issue #1 from bitbucket pull request...
Prepare github issue #2 from bitbucket pull request...
Prepare github issue #3 from bitbucket pull request...
Prepare github issue #4 from bitbucket pull request...
Traceback (most recent call last):
  File "./migrate-discussions.py", line 953, in <module>
    main()
  File "./migrate-discussions.py", line 949, in main
    bitbucket_to_github(bexport=bexport, gimport=gimport, cmap=cmap, args=args)
  File "./migrate-discussions.py", line 779, in bitbucket_to_github
    gissue_or_gpull = construct_gissue_or_gpull_from_bpull(bpull, bexport, cmap, args)
  File "./migrate-discussions.py", line 644, in construct_gissue_or_gpull_from_bpull
    issue_body = construct_gpull_request_body(bpull, bexport, cmap, args)
  File "./migrate-discussions.py", line 451, in construct_gpull_request_body
    sb.append(map_content(bpull["description"], cmap, args))
  File "./migrate-discussions.py", line 253, in map_content
    tmp = replace_implicit_links_to_prs(tmp, args)
  File "./migrate-discussions.py", line 109, in replace_implicit_links_to_prs
    return IMPLICIT_PR_LINK_RE.sub(replace_pr_link, body)
  File "./migrate-discussions.py", line 86, in replace_pr_link
    bpr_nr = int(match.group(2))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

User references are scrambled

In issue https://github.com/fpoli/viper-silicon/issues/367, in the "Nicolaj just closed..." comment by Malte, "@federico Poli" was tagged in the original comment (https://bitbucket.org/viperproject/silicon/issues/367/incompleteness-with-non-linear-arithmetic), but in the Github version, you see the string "{557058:2a067556-b105-4ce3-b3ec-bf9ecac58455}" instead.

Migrate commits from Mercurial to Git

For each non-merged pull request on bitbucket repository, pull the particular branch referenced in the pull request to a fresh branch (e.g. named bitbucket-pr-123) in the final repository.

Then, after migrating the repo we can open github pull requests for each non-merged bitbucket pull request, and github issues for merged pull requests.

For the migration: https://github.com/frej/fast-export

Migrate attachments

Migrating Pull Requests from Bitbucket to Github

Hi, first of all thank you for this amazing tool! Feel free to close this right away because it's not really an issue, but simply a question.

I'm test-driving to move one of my organisation's repositories to Github, but in the end all existing pull requests (already merged) are converted into issues. Is it meant to be like this?

I've been digging into the code and noticed that if a pull request is closed it automatically creates an issue (and closes it) on its behalf. I understand that actually creating a pull request that has already been merged is not trivial at all, just wanted to know if this is the default behaviour or if I'm doing something wrong :)

Thanks a lot!

Manual migration steps

• File logfile-0208-z3.zip of bitbucket issue viperproject/silicon/#208 is too big and cannot be uploaded as a gist file. This has to be done manually.
• Manually rewrite the .hgignore file to .gitignore (see #25).
• Publish the mapping of commits as a Github gist.
• Check Silver warnings:

Prepare github issue #314 from bitbucket pull request...
Warning: could not map mercurial commit '2a53e4eba176' (source of a PR) to git.
...
Prepare github issue #390 from bitbucket pull request...
Warning: could not map mercurial commit '7b83703b96ba' (source of a PR) to git.
...
Prepare github issue #418 from bitbucket pull request...
Warning: could not map mercurial commit '544d4e703501' (source of a PR) to git.
...
Prepare github issue #434 from bitbucket pull request...
Warning: could not map mercurial commit '90b3da501740' (source of a PR) to git.

• Check Silicon warnings:

Prepare github issue #425 from bitbucket pull request...
Warning: could not map mercurial commit '2b88aa4f08ee' (source of a PR) to git.
...
Prepare github issue #460 from bitbucket pull request...
Warning: could not map mercurial commit 'a0ac36b78221' (source of a PR) to git.
...
Prepare github issue #470 from bitbucket pull request...
Warning: could not map mercurial commit '8ece9dbc0928' (source of a PR) to git.

Unable to handle big files

This migration script is unable to handle big files, most likely due to GitHub. It is worth investigating if LFS is a choice and how to use it with "viperproject" account. Other alternatives should be considered otherwise.

This is affecting the following repositories:

• Sample Project / sample
• Sample Project / sample_qp
• Sample Project / sample_qp2

Migration of pull request

Migration of pull request https://bitbucket.org/viperproject/viperserver/pull-requests/7/debugger-support/diff is incomplete because no Git branch named aurecchia/viperserver/debugger-support is created by the scripts. The body of the Github pull request looks fine but the link points to nowhere:

Pull request 🔀 created by bitbucket user aurecchia on 2018-09-09 18:26
Last updated on 2019-04-23 16:42
Original Bitbucket pull request id: 7

Participants:

• @ignore_aterga (reviewer)
• bitbucket user aurecchia

Source: https://github.com/fpoli/viper-viperserver/commit/fb6266bde23243fada70f50074ac0eade28b7513 on aurecchia/viperserver/debugger-support
Destination: https://github.com/fpoli/viper-viperserver/commit/fdad473a306d89e0517d22767ad847b22d495772 on master
Marge commit: https://github.com/fpoli/viper-viperserver/commit/636c96f861687686e9b0af7b6cc2e44a3345fb0f

State: MERGED

Script is getting stuck when porting some repositories

For some reason the script is getting stuck while porting the following repositories:

• Deprecated / silicon-assertable
• Deprecated / silver-assertable
• Deprecated / silver-coarse
• Fork / silicon-coarse
• Fork / silicon-verifiedif
• Fork / silver-attributes
• Fork / silver-verifiedif

This seems to be related with the process of fetching forks of the repository to create local branches. More investigation is needed.

The filter for large file is not working properly in considering all files instead.

In Sample, all files seemed to be considered big, as all of them were mistakingly placed in LFS. Probably an error in Git LFS usage or in GitHub itself.

Embedded images in BitBucket images are not migrated

BitBucket supports embedded images in issue comments, which seem to be hosted on bitbucket. When an issue is migrated to GitHub, the URL is unchanged. If the BitBucket repo is public, I assume the images will appear to work in GitHub, at least until the BitBucket repo is deleted. However, if the BitBucket repo is private, the images do not show up in the GitHub issue.

The obvious solution would be to download the image from BitBucket, then somehow upload it to githubusercontent.com and link to it in the same way as images are handled in GitHub issues. However, according to this post, this might not yet be possible through the GitHub API.

Maybe another option would be create a new Gist, then clone the Gist as a repository and add the file to it, then push it back. The file should then be accessible via its raw URL, and a markdown link could be created in the issue to replace the BitBucket URL.

changes in issues not visible

In https://bitbucket.org/viperproject/carbon/issues/270/incomplete-behaviour-when-using-gotos-to#comment-51802030 one sees the bullet points showing what actions were taken by the users. In this case "changed title to Incomplete behaviour when using gotos to jump out of loops" and "edited description". In the Github issue, this is not the case: https://github.com/fpoli/viper-carbon/issues/270#issuecomment-583463321

I think this can be helpful and it would be nice to get this information if it's easily possible.

More links

Here are some examples of comments in which it would be possible to detect and convert to Github the branch name or Mercurial commit hash:

• "Although good progress was made in branch krantikiran_issue_216_bactrack_after_in, no solution that is general enough was reached yet" (Silver issue 216)
• "→ <<cset 3314d13def03>>" (Silver issue 38)
• "This issue was fixed in branch fabiopakk_issue_162_assume_as_inhale" (Silver issue 162)

Migration script is not handling Git repositories on BitBucket

Although most of the repositories in BitBucket are Mercurial, there are some Git repositories as well. The script ain't handling such cases.

pull request reviewer/assignees

Will reviewers/assignees of pull requests be migrated as well (this is not the case currently)?

Creation of issues silently fails

After creating roughly 251 Carbon issues, the creation of issues silently fails. Pull requests then start taking the IDs of non-created issues, breaking all links.

For example, https://github.com/fpoli/viper-carbon/issues/251 is an issue, however https://github.com/fpoli/viper-carbon/issues/252 is a (closed) pull request even if https://bitbucket.org/viperproject/carbon/issues/252 is an issue.

Port script is unable to handle wrong date

Some of the repositories have wrong date into their commits, which causes the exception bellow:

fatal: Invalid raw date "<devnull@localhost> 1439739323 +0200" in ident:  <> <devnull@localhost> 1439739323 +0200
fast-import: dumping crash report to .git/fast_import_crash_2931
Traceback (most recent call last):
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 529, in <module>
    notes=options.notes,encoding=encoding,fn_encoding=fn_encoding))
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 427, in hg2git
    sob,brmap,hgtags,encoding,fn_encoding)
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 248, in export_commit
    export_file_contents(ctx,man,added,hgtags,fn_encoding)
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 139, in export_file_contents
    strip_leading_slash(filename)))
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 37, in wr
    wr_no_nl(msg)
  File "/home/pakk/migration/fast-export/hg-fast-export.py", line 34, in wr_no_nl
    sys.stdout.write(msg)
IOError: [Errno 32] Broken pipe

This issues must be fixed in order to properly port all the repositories to GitHub.