iqss/redmine2github

Scripts to migrate redmine tickets to github issues

redmine2github

Scripts to migrate redmine tickets to github issues. This is for a 1-time move--so it's a mix of automation and manual decisions. e.g., Get-it-done, but make the process repeatable.

Setup with virtualenvwrapper

• mkvirtualenv redmine_move
• pip install -r requirements/base.txt

Make a config file

• Copy "/src/settings/local_sample.py to "/src/settings/local.py" and fill in variables
• Fill in your redmine and github credentials
  • For the Redmine API key, see "You can find your API key on your account page..."
  • For github, enter your credentials. (To do, use a Personal API token)

Workflow

(1) Download your open redmine issues

• Each issue is saved as a .json file, including relevant "children", "journals", "watchers", "relations"
  • The file naming convention is by ticket issue number.
    • e.g. Issue 387 becomes "00387.json"
  • Files are saved to the following directory:
    • from settings/local.py: (REDMINE_ISSUES_DIRECTORY)/(current date in 'YYYY_MMDD' format)
      • e.g. ../working_files/redmine_issues/2014-0709/(json files here) ../working_files/redmine_issues/2014-0709/03982.json ../working_files/redmine_issues/2014-0709/04050.json
• A json dict is saved to "issues_list.json" that maps the redmine issue number to the issue subject. For example: * e.g. ../working_files/redmine_issues/2014-0709/issues_list.json

    {
        "03982": "Create Genomics Metadata Block", 
        "04050": "Additional Astronomy FITS File Metadata Support for Units", 
        "04051": "Metadata: Astronomy - Changes to Ingest and Display of Resolution Elements", 
        "04072": "Edit Dataverse: Checking/unchecking use facets from Host Dataverse undoes any changes in the rest of the form."
    }
    

Example of downloading redmine issues

• cd into the src/redmine_ticket directory
• update the bottom of the "redmine_issue_downloader.py" file
• Currently looks something like this:

if __name__=='__main__':
    from settings.base import REDMINE_SERVER, REDMINE_API_KEY, REDMINE_ISSUES_DIRECTORY
    #rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 'dvn', REDMINE_ISSUES_DIRECTORY)
    rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 1, REDMINE_ISSUES_DIRECTORY)
    rn.download_tickets()

• run it:

../redmine2github/src/redmine_ticket> python redmine_issue_downloader.py

(2) Migrate your issues to a github repository

---

Note 1: Once added, a github issue cannot be deleted. Therefore, to test your migration, create a new "scratch" github repository. Once you're satisfied that the migration works, delete the scratch repository.

---

---

Note 2: The current GitHub API limit is 5,000/day. Adding each issue may have 1-n API calls. Plan appropriately.

• 1 API Call: Create issue with labels, milestones, assignee
  • This process creates a json file mapping { Redmine issue number : GitHub issue number}
• 0-n API Calls for comments: A single API call is used to transfer each comment
• 2 API Calls for related issues (optional): After all issues are moved
  • Call 1: Read each GitHub issue
  • At the bottom of the description, use the Redmine->GitHub issue number mapping to add related issue numbers and child issue numbers
  • Call 2: Update the GitHub description

---

Quick script

• cd into the src/github_issues directory
• update the bottom of the "migration_manage.py" file
• Currently looks something like this:

if __name__=='__main__':
    # e.g. json_input_directory, where you downloaded the redmine JSON
    #      json_input_directory="some_dir/working_files/redmine_issues/2014-0709/"
    #
    json_input_directory = os.path.join(REDMINE_ISSUES_DIRECTORY, '2014-0702')

    kwargs = dict(include_comments=True\
                , include_assignee=False\
                , redmine_issue_start_number=4123\
                , redmine_issue_end_number=4134\
                , user_mapping_filename=USER_MAP_FILE       # optional
                , label_mapping_filename=LABEL_MAP_FILE     # optional
                , milestone_mapping_filename=MILESTONE_MAP_FILE # optional
            )
    mm = MigrationManager(json_input_directory, **kwargs)
    mm.migrate_issues()

• run it:

../redmine2github/src/github_issues>python migration_manager.py

Label Map Notes

The label map is optional. It allows you to assign label names and colors by creating a label map file.

• See Sample Label Map, sample_label_map.csv

---

• The redmine_type column is for user convenience only, it is ignored by the program. So watch out for name collisions

---

• Pertains to Redmine name values in fields status, tracker, priority, or custom_fields
• If no map is specified in the MigrationManager kwargs:
  • The status, tracker, priority, or custom_fields names in Redmine issues are made into GitHub labels. See "def get_label_names" in the label_helper.py file
    • A Redmine status of "New" would turn into label "Status: New"
    • A Redmine tracker of "Feature" would turn into label "Tracker: Feature"
    • A Redmine priority of "Urgent" would turn into label "Priority: Urgent"
    • A Redmine custom field of "UX/UI" turns into label "Component: UX/UI"
  • If no map is specified, then newly created labels will not have a color

Map Notes - How is the map used?

• The map is specfied in the settings/local.py file under LABEL_MAP_FILE
• If a status, tracker, priority, or custom_field name in a Redmine ticket is NOT found in the map, that name value will NOT be moved to GitHub
• The map file is "dumb." If you would like to map more than one status name to a single status label, simply repeat it.
  • In the example below, the "redmine_name"s "In Design" and "In Dev" are both mapped to the label named "Status 3: In Design/Dev"
  • For repeated github_label_names, make sure they're the same:)

redmine_type, redmine_name, github_label_name, github_label_color
status, In Design, Status 3: In Design/Dev,996600
status, In Dev, Status 3: In Design/Dev,996600

• When the map is read, the values are trimmed. e.g. ", In Design ," would become "In Design" with leading/trailing spaces removed

redmine2github's Issues

Strings of Redmine user names should be GitHub usernames prepended with @ instead

For example, at https://github.com/raprasad/test-issue-migrate/issues/39 rather than seeing

Author Name: Philip Durbin

it should show

Author Name: @pdurbin

because it has a useful link per https://help.github.com/articles/writing-on-github#name-and-team-mentions-autocomplete

This will require us to come up with a mapping of people's names in Redmine vs. GitHub.

Can't get issue count when redmine requires authentication

This is looking really promising, but the redmine issue downloader isn't working for me - I just get a 403 error.

I've found this is because the issue count is downloaded using urllib2 rather than the Redmine client, and the script doesn't set any credentials for this request. Our redmine seems to requires authentication for the API, so this doesn't work.

I'm not sure the best way to set this authentication for the request, or indeed why this is done differently to the other Redmine requests which seem to be working OK.

Requested resource doesn't exist

When I try and follow the directions in the readme, I get:

  File "/home/dmd/.environments/redmine_move/local/lib/python2.7/site-packages/redmine/__init__.py", line 88, in request
    raise ResourceNotFoundError
redmine.exceptions.ResourceNotFoundError: Requested resource doesn't exist

Any suggestions?

How to get only issues of a certain project downloaded from Redmine?

How do you get only the issue json files relating to one project from Redmine?

If Redmine separates issues to folders only by date, that seems a bit problematic.

Facing ''Github' object has no attribute 'issues' when run migration_manager.py

Traceback (most recent call last):
File "/Users/akshayjoshi/Developement/redmine2github/src/github_issues/migration_manager.py", line 228, in
mm.migrate_issues()
File "/Users/akshayjoshi/Developement/redmine2github/src/github_issues/migration_manager.py", line 200, in migrate_issues
github_issue_number = gm.make_github_issue(json_fname_fullpath, **gm_kwargs)
File "/Users/akshayjoshi/Developement/redmine2github/src/github_issues/github_issue_maker.py", line 360, in make_github_issue
issue_obj = self.get_github_conn().issues.create(github_issue_dict)
AttributeError: 'Github' object has no attribute 'issues'

python migration_manager.py Error

when I run python migration_manager.py, output as below, my server is local gitlab setup by myself.
any suggestion?

Skipping Redmine issue: 65 (start at 66)

(1) Loading redmine issue: [66] from file [00066.json]

Attempt to create issue: [#66][wangwang]
Milestone: hades5.0
Traceback (most recent call last):
File "migration_manager.py", line 228, in
mm.migrate_issues()
File "migration_manager.py", line 200, in migrate_issues
github_issue_number = gm.make_github_issue(json_fname_fullpath, *_gm_kwargs)
File "/home/jielun/redmine2github-master/src/github_issues/github_issue_maker.py", line 345, in make_github_issue
milestone_number = self.milestone_manager.get_create_milestone(rd)
File "/home/jielun/redmine2github-master/src/github_issues/milestone_helper.py", line 181, in get_create_milestone
milestone_number = self.get_create_milestone_number(mstone_name)
File "/home/jielun/redmine2github-master/src/github_issues/milestone_helper.py", line 106, in get_create_milestone_number
mnum = self.get_mile_stone_number(title)
File "/home/jielun/redmine2github-master/src/github_issues/milestone_helper.py", line 136, in get_mile_stone_number
for page in milestones:
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/base.py", line 100, in next
return self.next()
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/smart.py", line 91, in next
if self.page <= self.pages:
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/smart.py", line 101, in pages
return self.getter.last
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/smart.py", line 47, in last
self()
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/smart.py", line 16, in wrapper
return func(self, page)
File "/usr/lib/python2.7/site-packages/pygithub3/core/result/smart.py", line 39, in call
response = self.method(page=page)
File "/usr/lib/python2.7/site-packages/pygithub3/core/client.py", line 84, in get
response = self.request('get', request, *_kwargs)
File "/usr/lib/python2.7/site-packages/pygithub3/core/client.py", line 71, in wrapper
return func(self, verb, request, **kwargs)
File "/usr/lib/python2.7/site-packages/pygithub3/core/client.py", line 80, in request
GithubError(response).process()
File "/usr/lib/python2.7/site-packages/pygithub3/core/errors.py", line 35, in process
raise raise_error()
File "/usr/lib/python2.7/site-packages/pygithub3/core/errors.py", line 20, in error_404
raise NotFound("404 - %s" % self.debug.get('message'))
pygithub3.exceptions.NotFound: 404 - Not Found