mavam/trac-hub

:bookmark: Converts trac tickets to github issues.

trac-hub's Issues

Dependencies do not build with Ruby 2.7

I'm on Pop!_OS 20.04 (like Ubuntu 20.04) with Ruby 2.7 and both mysql2 and sqlite3 failed during bundle install. I was able to work around the issue by installing the ruby-mysql2 and ruby-sqlite3 packages distributed by Ubuntu's repositories instead of installing them with bundle/gem. I'm not a ruby person so there might be a more straightforward way to fix this.

Related: brianmario/mysql2#1113

Implement rate-limiting for each github account

Make sure that each instance of a GitHub client does not exceeds its rate limit, which would abort the migration process. To this end, weave in a check before each member function of the GitHub class.

Markdownify not behaving correctly

Given the string

In [changeset:5851]:
{{{
#!CommitTicketReference repository="" revision="5851"
Fixed #2137: Race condition in 183 re-transmission can result in a deadlock.
}}}

markdownify returns (ignore the commit hash - fake revmap for debugging purpose)

> In [[r5851](../commit/99999)](../commit/99999):
> Fixed #2137: Race condition in 183 re-transmission can result in a deadlock.
>

It seems there's a double replacement.

markdownify does not behave conrrectly - 2

• it merges the header of first level with the adjacent paragraph
• it converts it all to a block quote

as an example to indicate what happens on my project:

= Anforderung =

Im Prinzip gehe ich im meinem Bereich auf „Mitarbeiter (mit E-Mail) hinzufügen“ und gebe dort nur ein Buchstaben ein. Daraufhin tauchen rechts dann alle Mitarbeiter auf, bei denen der Buchstabe mit vorkommt. Dies funktioniert mit allen Buchstaben. Es werden jedoch auch alle weiteren Infos der Mitarbeiter wie Adresse, Telefonnummer usw. angezeigt.

Natürlich können die Daten nur von „Bereichsleitern“ und nicht von jedermann eingesehen werden, wollte die Info trotzdem an euch weitergeben.

= Lösungsansatz =

das ist natürlich eine Sicherheitslücke
Lösung:
1.  wenn mehr als drei treffer sind, dann wird nichts angezeigt, sondern der Hinweis, dass die Suchkriterien verfeinert werden müssen.
2. Wir zeigen die Telefonnummern nicht mehr an.
3. es müssen mindestes drei buchstaben im Suchkriterium stehen.


= Aufgaben =

Lösung 1 implemeniteren

= Testen =

auf madb1 in irgendeinen Bereich gehen, dort bei email den buchstaben "t" eingeben,

== erwartetes Ergebenis ==

es kommt die Meldung "Zu Viele Mitarbeiter"

converts to

> 
> # AnforderungIm Prinzip gehe ich im meinem Bereich auf „Mitarbeiter (mit E-Mail) hinzufügen“ und gebe dort nur ein Buchstaben ein. Daraufhin tauchen rechts dann alle Mitarbeiter auf, bei denen der Buchstabe mit vorkommt. Dies funktioniert mit allen Buchstaben. Es werden jedoch auch alle weiteren Infos der Mitarbeiter wie Adresse, Telefonnummer usw. angezeigt.
> 
> Natürlich können die Daten nur von „Bereichsleitern“ und nicht von jedermann eingesehen werden, wollte die Info trotzdem an euch weitergeben.
> 
> # Lösungsansatzdas ist natürlich eine Sicherheitslücke
> Lösung:
> 1.  wenn mehr als drei treffer sind, dann wird nichts angezeigt, sondern der Hinweis, dass die Suchkriterien verfeinert werden müssen.
> 2. Wir zeigen die Telefonnummern nicht mehr an.
> 3. es müssen mindestes drei buchstaben im Suchkriterium stehen.
> 
> 
> # AufgabenLösung 1 implemeniteren
> 
> # Testenauf madb1 in irgendeinen Bereich gehen, dort bei email den buchstaben "t" eingeben,
> 
> ## erwartetes Ergebenis
> 
> es kommt die Meldung "Zu Viele Mitarbeiter"
> 
>   

customize markdownify()

• would be nice if markdowinify() were customizable
• actually, what I really want is to parse the wiki/InterMapTxt file from the db and use that 👍

Invalid filter expression

Hi everyone

I got the script running but it immediately stops with an exception.

trac:/usr/src/trac-hub/trac-hub# ./trac-hub
[2021-03-04 10:53:51] INFO | loading tickets
[2021-03-04 10:53:51] INFO | Last created issue on GitHub is '0'
[2021-03-04 10:53:51] INFO | migrating issues
[2021-03-04 10:53:51] INFO | creating issue 1 (dakapo)
Traceback (most recent call last):
10: from ./trac-hub:476:in '

'
9: from ./trac-hub:76:in 'migrate'
8: from ./trac-hub:109:in 'migrate_tickets'
7: from ./trac-hub:109:in 'each'
6: from ./trac-hub:127:in 'block in migrate_tickets'
5: from ./trac-hub:177:in 'compose_issue'
4: from ./trac-hub:261:in 'ticket_change'
3: from ./trac-hub:85:in 'trac_mail'
2: from /var/lib/gems/2.5.0/gems/sequel-5.42.0/lib/sequel/dataset/query.rb:1047:in 'where'
1: from /var/lib/gems/2.5.0/gems/sequel-5.42.0/lib/sequel/dataset/query.rb:1274:in 'add_filter'
/var/lib/gems/2.5.0/gems/sequel-5.42.0/lib/sequel/dataset/query.rb:1307:in 'filter_expr': Invalid filter expression: ["name = "email" AND sid = ?", "dakapo"] (Sequel::Error)

Is anybody out there that can help me with this issue?

tools should be implemented in ruby and use config.yaml

I am running a docker based setup (#26) where mysql is in a separate container, so I do not have msql console available. Therefore the scripts in tools do not work.

I propose to integrate the tools to trac-hub itself.

support github enterprise

where SECRETHOSTNAME should be a variable or something.

   def initialize(user, pass, repo)
     @user = user
     $logger.debug("#{@user} connects to github repo '#{repo}'")
+    Octokit.configure do |c|
+      c.api_endpoint = "https://SECRETHOSTNAME/api/v3/"
+    end

details: https://github.com/octokit/octokit.rb#working-with-github-enterprise

support hashed attachments

attachments are stored in hashed format in the trac data dir now. maybe make this a switch.

diff --git a/trac-hub b/trac-hub
index daece2b..657260a 100755
--- a/trac-hub
+++ b/trac-hub
@@ -9,6 +9,8 @@ require 'yaml'
 require 'set'
 require 'singleton'
 require 'uri'
+require 'digest/sha1'
+

 class GracefulQuit
   include Singleton
@@ -341,7 +343,19 @@ class Migrator
           when 'attachment'
             text = "**Attachment added:**"
             if @attachurl
-              url="#{@attachurl}/#{i}/#{ticket[:filename]}"
+              #hashed URLs.   
+              #ticket 14/blahblah.java is in 
+              #  `fa3/fa35e192121eabf3dabf9f5ea6abdbcbc107ac3b/d8be6619342d0340df06a23e8a9f2b720ac32bc8.java`
+              #  <first 3 of # hash>/<full # hash>/<filename hash>.<extension>
+              hash_dir = Digest::SHA1.hexdigest("#{i}")  # 14 -> fa35e192121eabf3dabf9f5ea6abdbcbc107ac3b
+              hash_par = hash_dir[0,3] # 14 -> fa3
+              hash_file = Digest::SHA1.hexdigest("#{ticket[:filename]}") # -> d8be6619342d0340df06a23e8a9f2b720ac32bc8
+              file_suff = File.extname(ticket[:filename])
+              url="#{@attachurl}/#{hash_par}/#{hash_dir}/#{hash_file}#{file_suff}"
+
+              # OTHER WAY (non hashed)
+              #url="#{@attachurl}/#{i}/#{ticket[:filename]}"
+
               if [".png", ".jpg", ".gif", ".PNG", ".JPG", ".GIF"].include? File.extname(ticket[:filename])
                 text = "#{text} `#{ticket[:filename]}` (#{(ticket[:size]/1024.0).round(1)} KiB)\n![#{ticket[:filename]}](#{URI.escape(url)})"

specify a custom media type in the 'Accept' header

I see links in the console which end up in messages like this. What does this mean?

{
  "message": "If you would like to help us test the Issue Importer API during its preview period, you must specify a custom media type in the 'Accept' header. Please see the docs for full details.",
  "documentation_url": "https://gist.github.com/jonmagic/5282384165e0f86ef105#check-status-of-issue-import"
}

dockerify trac-hub

First of all I need to express that trac-hub is awesome. After spending some days with tracboat and other scripts this is the first stuff which ended in promising results at github (maybe because it is written ruby :-)

This morning I played with trac-hub. I used docker for that:

FROM ruby:2.5

# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1

WORKDIR /usr/src/app

COPY Gemfile Gemfile.lock ./
RUN bundle install

COPY . .

ENTRYPOINT cd /usr/src/myapp/trac-hub && bash

I start it with docker-compose

  mysql:
    image: mysql:5.7
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=password
    ports:
      - 3306:3306
    volumes:
      - mysql:/var/lib/mysql

 trac-hub:
    image: bwl21/trac-hub
    build: trac-hub

    restart: unless-stopped

    volumes:
      - .:/usr/src/myapp
      - /Users/beweiche/gitlab/data:/var/opt/gitlab
    links:
      - mysql
    depends_on:
      - mysql

Inovoke as

docker-compose run --rm trac-hub

keep github ids in sync with trac ids even if trac has no consecutive ticket ids

Request

By whatever reason we have deleted a bunch of tickets in our trac. I now would like to migrate trac into a fresh github project. Is there a way to ensure that the id in github are the same as in trac, even if the ids are not consecutive.

This is the opposite case than the one when we do in an exisitng github repository which already has issues.

Solution approach

• We could create "dummy tickets" for the missing ids and delete the same afterwards.
• rewrite the ticket references in comments - but this is not possible for commit messages.

failed to create milestone: already exists

trac-hub seemed to only load about 28 milestones or so. Milestone icu-3.5 already existed, but wasn't in the @milestones list apparently.

[2016-08-24 16:55:15] INFO  | migrating milestones
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.0'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.0.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.0.2'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.0.3'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.2'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.2.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.4'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.6'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.6.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.6.2'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-2.8'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.0'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.2.0.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.2.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.2'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.3'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.4'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.4.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.4.5'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-1.6'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-1.7'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-1.8'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-1.8.1'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.2'
[2016-08-24 16:55:15] WARN  | skipping already existing milestone 'icu-3.6'
[2016-08-24 16:55:15] INFO  | creating new milestone icu-3.5
/usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/response/raise_error.rb:16:in `on_complete': POST https://github.ibm.com/api/v3/repos/srloomis/test-gcoctrac/milestones: 422 - Validation Failed (Octokit::UnprocessableEntity)
Error summary:
  resource: Milestone
  code: already_exists
  field: title // See: https://developer.github.com/enterprise/2.7/v3/issues/milestones/#create-a-milestone
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/response.rb:9:in `block in call'
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/response.rb:57:in `on_complete'
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
    from /usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
    from /usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/middleware/follow_redirects.rb:61:in `call'
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
    from /usr/local/lib/ruby/gems/2.3.0/gems/faraday-0.9.1/lib/faraday/connection.rb:177:in `post'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sawyer-0.6.0/lib/sawyer/agent.rb:94:in `call'
    from /usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/connection.rb:154:in `request'
    from /usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/connection.rb:28:in `post'
    from /usr/local/lib/ruby/gems/2.3.0/gems/octokit-4.1.1/lib/octokit/client/milestones.rb:52:in `create_milestone'
    from ./trac-hub:512:in `create_milestone'
    from ./trac-hub:134:in `block in migrate_milestones'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/dataset/actions.rb:139:in `block in each'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:334:in `block (2 levels) in fetch_rows'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/resultset.rb:138:in `each'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:325:in `block in fetch_rows'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:269:in `query'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:185:in `block (2 levels) in _execute'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/database/logging.rb:33:in `log_yield'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:185:in `block in _execute'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/database/connecting.rb:249:in `synchronize'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:178:in `_execute'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:128:in `execute'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/dataset/actions.rb:921:in `execute'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/adapters/sqlite.rb:319:in `fetch_rows'
    from /usr/local/lib/ruby/gems/2.3.0/gems/sequel-4.26.0/lib/sequel/dataset/actions.rb:139:in `each'
    from ./trac-hub:112:in `migrate_milestones'
    from ./trac-hub:86:in `migrate'
    from ./trac-hub:735:in `<main>'

ticket creation fails

This is my first try with the script. I installed all dependencies and get this result:

bundle exec trac-hub 
/home/behrisch/trac-hub-master/vendor/bundle/ruby/2.4.0/gems/sequel-4.26.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/home/behrisch/trac-hub-master/vendor/bundle/ruby/2.4.0/gems/sequel-4.26.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
[2017-07-04 08:12:38] INFO  | loading tickets
[2017-07-04 08:12:38] INFO  | Last created issue on GitHub is '0'
[2017-07-04 08:12:38] INFO  | migrating issues
[2017-07-04 08:12:39] INFO  | creating issue 1 (testuser)
bundler: failed to load command: trac-hub (trac-hub)
NoMethodError: undefined method `[]' for nil:NilClass
  trac-hub:168:in `compose_issue'
  trac-hub:118:in `block in migrate_tickets'
  trac-hub:102:in `each'
  trac-hub:102:in `migrate_tickets'
  trac-hub:69:in `migrate'
  trac-hub:435:in `<top (required)>'

bundler

for the non python savvy…  needed to do the following:

$ gem install bundler
$ bundler install