moimael/trac-to-gitlab

Easily migrate your Trac instance to Gitlab

What

This script migrates milestones, issues and wiki pages from trac to GitLab.

NOTE: This project is in deprecated, in favor of tracboat/tracboat.

Features

• Component & Issue-Type are converted to labels
• Comments to issues are copied over
• Supports two modes of tansfer:
• Using GitLab web API
• Direct access through GitLab's database and file system
• In direct mode, attachments are transfered and the issues and notes dates and ownership are preserved
• In API mode, attachments are not transfered, issues and notes are owned by a single user and their dates are the current date.

How

Migrating a trac project to GitLab is a relatively complex process involving four steps:

• Create a new project
• Migrate the repository (can just be cloning a git repository if the trac project is already using git or could involve converting from subversion using git-svn)
• Create the users for the project
• Migrate issues and milestones
• Migrate wiki pages

This script takes care of the last two bullet points and provides help for the third one.

Usage:

1. copy migrate.cfg.example to migrate.cfg
2. configure the values
3. run ./collect-users.py to extract the user names from Trac
4. update migrate.cfg and create the users in GitLab
5. run (./migrate.py). Make sure you test it on a test project prior, if you run it twice against the same project you will get duplicated issues unless you're using direct access with overwrite set to yes.

Issues and milestones are copied to GitLab.

Wiki pages are copied to a folder on your machine and must be pushed into GitLab using wiki's git access.

GitLab versions

The database model should correspond to the version of GitLab that you are using.

This repo contains models for multiple versions (gitlab_direct/model.py) and the version number should be updated correspondingly in the imports in gitlab_direct/init.py and gitlab_direct/Connection.py.

To support a new version, use pwiz.py:

$ pwiz.py -e postgresql -u gitlab gitlabhq_production > gitlab_direct/model<version>.py

Manual updates must then be applied, see for instance the manual updates for 6.4.

Configuration

The configuration must be located in a file named "migrate.cfg"

Source

• url - xmlrpc url to trac, e.g. https://user:[email protected]/projects/thisismyproject/login/xmlrpc

Target

• project_name - the destination project including the paths to it. Basically the rest of the clone url minus the ".git". E.g. jens.neuhalfen/task-ninja.
• method - direct or api

ÄPI mode:

• url - e.g. https://www.exmple.com/gitlab/api/v3
• access_token - the access token of the user creating all the issues. Found on the account page, e.g. secretsecretsecret
• ssl_verify - set to yes to verify SSL server certificates.

Direct mode:

• ```overwrite````- if set to yes, the milestones and issues are cleared for this projects and issues are recreated with their trac id (useful to preserve trac links)
• db-name - MySQL database name
• db-user - MySQL user name
• db-password - MySQL password
• uploads - GitLab uploads directory
• usernames Comma separed list of username mappings such as: trac1->git1, trac2->git2

Wiki

• migrate - Should the wiki pages be converted?
• target-directory - Directory in which the wiki pages should be written

Issues

• migrate - Should we migrate issues and milestones?

Licenses

LGPL license version 3.0 (see the licences directory).

History

• The main program has been cloned from https://gitlab.dyomedea.com/vdv/trac-to-gitlab which itself has been cloned from https://github.com/neuhalje/hack-copy-track-issues-to-gitlab
• Trac2down.py (the conversion of trac wiki markup to markdown) has been cloned from https://gist.github.com/sgk/1286682 and https://gist.github.com/tcchau/4628317

Requirements

• Python 3.2, xmlrpclib, requests
• Trac with XML-RPC plugin enabled
• Gitlab

And also, if you use the direct access to GitLab's database:

• peewee
• PyMySQl

trac-to-gitlab's Issues

Error while importing issues from trac to gitlab with API version v4

While migrating my trac issues to gitlab isung the gitlab v4 api i encountered an error regarding the Ticket ids.
Gitlab uses two parameters, id für global ticket count and iid for project-specific ticket count.

Because of that the line 109 in the Connection.py file should be altered as follow:
new_ticket_id = new_ticket["id"]
should be replaced by
new_ticket_iid = new_ticket["iid"]

Please archive this repo

Hi @moimael 👋 The deprecation note is a good 1st step, but please also consider GitHub's archive option for this repo, to make it even clearer, that it's no longer maintained. (Context) Thank you!

can't get direct mode to work with postgres

I'm using the Bitnami Gitlab CE image, which runs Gitlab on PostgreSQL. I can import from Trac to there fine using the API mode, but as is documented in the README, that doesn't import attachments, issue owners, or issue dates. So I want to use direct mode instead.

So far I can't get it to work. No matter what settings I use, when I run migrate.py I get Peer authentication failed for user "gitlab" (or for user "bitnami", etc.). The instructions for direct mode in migrate.cfg say:

You need to create a PostgreSQL user for the user that runs the migration and map it to the gitlab user in /var/opt/gitlab/postgresql/data/pg_ident.conf

I've tried many permutations of this, setting variations of lines like

gitlab bitnami postgres

in /var/opt/gitlab/postgresql/data/pg_ident.conf, and restarting Postgres (/opt/bitnami/ctlscript.sh restart postgresql). Still I get Peer authentication failed.

I guess I'm confused about who exactly "the user that runs the migration" (presumably me, user bitnami) and "the gitlab user" are. Further guidance and examples would really help here.

Thank you.

Thank you!

Hi, I would like to thank you for all the inspiration that this project gave me. In the near future I'm going to face the problem of migrating a lot (hundreds) of Trac instances over to GitLab and in the attempt to make the trac-to-gitlab code more easy to automate I think I went too far with the cleanup and refactoring process., so I decided to switch over to a brand new project, actually keeping the trac-to-gitlab refactored code as the core data extraction and migration logic. You can find the project here:

https://github.com/nazavode/tracboat

I gave it the form of a command line tool, with subcommands (a la git) and the ability to export a Trac project to a local file (in multiple human readable formats) and to migrate (a previously exported project or directly from Trac) to a mock GitLab instance (into a SQLite database) just to be able to inspect it before the actual migration in production. I kept the GPLv3 because I think I had to inherit it to be compliant and I mentioned you in the credits (in the README.rst and in the AUTHORS file as well). If you think that someone else should be mentioned please let me know or just open a pull request. Of course, code reviews, issues and pull requests are highly appreciated!

Thank you again!

preserving ticket numbers

i was looking for trac to gitlab converter so the ticket references in commit messages would map to proper ticket

but this converter (api mode) creates the ticket in some random order.

also in trac i had multiple svn projects relating to single trac instance, while in gitlab i have each of those git projects as separate gitlab projects, so i would need to filter out what tickets to import (by component name perhaps, or just manual listing. i have ~120 tickets i want to preserve historically

collect-users.py gives "TypeError: __init__() takes exactly 6 arguments (5 given)"

~/E/E/w/trac-to-gitlab> ./collect-users.py
Traceback (most recent call last):
  File "./collect-users.py", line 110, in <module>
    dest = Connection(db_name, db_user, db_password, uploads_path)
TypeError: __init__() takes exactly 6 arguments (5 given)

This is in Cygwin x86_64. db_name, db_user, db_password, and uploads_path are all at their default values in migrate.cfg. I did run pip install -r requirements.txt.

"Labels" not properly migrated

The script worked great allowing me to migrate from Trac 0.11.7 to Gitlab 8.14.1.
All the issues and wiki migrated ok.

But now that I'm using Gitlab, the "Labels" that were transferred from Trac to Gitlab seem to be in a weird state.
They show up in the existing issue list, but I can't assign those labels to any new issues. Also, they don't show up in the list of Labels for the project.
If I try to create a new label with the same name it gives an error "Title has already been taken".

xml.parsers.expat.ExpatError: not well-formed (invalid token)

I ran the users.sh and I'm only getting to ticket #3735 out of 18,000+ tickets. I assume there is a character in the changelog that the xml parser doesn't like. When I Googled the error, I see that many times it's a '&' that xml parsers dislike, but I have none of those. I do see a ' < ' in one comment (surrounded by a single whitespace on both sides). I scoured documentation here and with tracboat and see no mention of xml parsing errors or how to handle them.

........
ticket id: 3733
ticket index: 3732
owner: joeschmo3
reporter: joeschmo2
ticket message poster:  joeschmo1
ticket message poster:  joeschmo2
ticket message poster:  joeschmo1
ticket id: 3734
ticket index: 3733
owner: joeschmo3
reporter: joeschmo2
ticket message poster:  joeschmo1
ticket message poster:  joeschmo2
ticket id: 3735
ticket index: 3734
owner: joeschmo4
reporter: joeschmo4
Traceback (most recent call last):
  File "./collect-users.py", line 127, in <module>
    collect_users(source)
  File "./collect-users.py", line 98, in collect_users
    changelog = source.ticket.changeLog(src_ticket_id)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1283, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1316, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1487, in parse_response
    p.feed(data)
  File "/usr/lib/python2.7/xmlrpclib.py", line 558, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 152, column 97

Additional question: does trac_to_git look only at the current state of changelog entries or does it also look at previous versions of modified changelog comments? If it's the latter, that's a bit more painful for debugging.

Unable to import from a Trac server with an invalid certificate

Import fails with SSL Certificate verify failed when Trac server does not have a valid certificate. This is different that the config-controllable ssl_verify config option.