synthead/github-migrator

A tool for migrating Bitbucket issues into GitHub as GitHub issues

github-migrator's Issues

Validate configuration data

Right now, configuration data is being passed to the issue handlers verbatim. If data is malformed, the library the handler uses will raise an exception when an error is received from the server, but some mild validation should be added to reduce requests and provide a better user experience.

Migrate comments

As of right now, this tool only migrates the original issue without the comments. Migrate the comments, too.

Extend unit tests to cover exceptions from issue handlers

Since the issue handler exceptions are being handled in github-migrator, it would be a good idea to write some tests to ensure that these exceptions are being raised appropriately.

Add options for choosing the source and destination issue handlers

Most of the codebase is designed from the ground-up to be modular in design, for the time being, it's hard-coded to read from Bitbucket issues read write to GitHub issues. We'll want to add some options for allowing the user to pick where to read from and what to write to. These options should be available in the configuration file, the command-line options, and when the user is prompted.

Keeping the modular design in mind, there will need to be some logic to specify issue handler classes from a user's String configuration. It is very important that this logic is not blindly accepted as this could lead to loading erroneous classes as issue handlers. This would cause very strange bugs, and possibly be a security concern, especially if the codebase is used behind a webserver in the future, for example. These problems are easily mitigated by generating a whitelist map of issue handlers (in the IssueHandlers module) based off their handler_name attributes.

Parse Confluence markup and translate it to markdown for GitHub

As of right now, the issue body data is being directly copied to GitHub issues. This includes Confluence markup that is not understood by GitHub, so visible code will be displayed in GitHub issue bodies if present in Bitbucket issues.

It looks like there isn't a lot of support for parsing confluence syntax, so it might be necessary to build a gem for this. Kramdown would be a great library to build this gem with.

Use multiple threads for faster issue creation

Currently, one issue is created at a time, which might prove to be too slow when migrating a large number of issues. The speed of the import can be greatly improved by using multiple threads.

We'll want to keep an eye out for API rate limiting. If too many requests are made to an issue destination at once, we'll likely see errors being returned, so we'll want to catch these errors and write some logic for them. For this matter, we should also have client-side rate limiting to ensure that only a certain amount of requests are being made at a time.

Migrate issues as the original user

Not sure if this is possible, but it would be super handy to have the GitHub issues be created as the original user from Bitbucket. This would likely require some sort of superuser access to the GitHub API, and would likely involve creating users along the way. This would be pretty easy if GitHub and Bitbucket were both LDAP-enabled, however, which is very common in the enterprise.

Issue handlers should be able to gather their own configuration data

At the moment, lib/config_data.rb is hard-coded to gather configuration specifically for Bitbucket and GitHub. This is not a modular approach and would encourage complexity if more issue handlers are added.

It would still be a good idea to still keep a class for common logic, but the actual data being gathered should be a responsibility of the handlers themselves. This includes knowing what data to read from configuration files, what options to parse from the command line, and what questions to ask the user for remaining data. In addition, the --help text should also include all the options for these modular sources.

Add tests for gathering configuration data

Since the configuration data gathering has grown to be more comprehensive, we'll want to include some tests to ensure that it's behaving the way it should. At a minimum, these tests should be included:

• No error when configuration file doesn't exist
• Command line options override configuration file settings
• User is prompted for configuration that was not found from command line or configuration file settings

Refactor lib/config_data.rb into multiple files for user input, command line options, and configuration file parsing

Originally, a configuration file was the only source for configuration, but that has exanded to command-line options and prompting for user input. This has caused lib/config_data.rb to bloat quite a bit and it's getting harder to read.

Create a new directory in lib/ for parsing configuration, and split up lib/config_data.rb into three files for user input, command line options, and configuration file parsing. The behavior of ConfigData should remain the same: all three sources should continue to be combined into one unified configuration like it is now.

Wrap codebase into modular gems

Put the codebase into a few modular gems so it can be used in other projects. The framework itself could be a "base" gem (github-migrator), and the endpoints could be abstracted into "support" gems to provide specific functionality (github-migrator-bitbucket).

We'll also want to keep the github-migrator executable as part of the base github-migrator gem. This way, a user can simply run gem install github-migrator and run github-migrator on their system.

Add support for dynamic multi-threaded issue creation rate limiting

Instead of hard-coding a maximum number of simultaneous requests for the destination's API, use a little logic to increase the number of requests until the server starts to incur rate limiting, then back the requests down a little. This'll allow for the fastest issue creation possible as permitted from the server.