Giter Site home page Giter Site logo

svn-all-fast-export / svn2git Goto Github PK

View Code? Open in Web Editor NEW
256.0 14.0 101.0 308 KB

:octopus: A fast-import based converter for an svn repo to git repos

License: GNU General Public License v3.0

QMake 0.55% C++ 70.70% Dockerfile 0.26% Shell 28.48%
command-line-tool git subversion svn svn2git version-control

svn2git's Introduction

svn-all-fast-export aka svn2git

This project contains all the tools required to do a conversion of an svn repository (server side, not a checkout) to one or more git repositories.

This is the tool used to convert KDE's Subversion into multiple Git repositories. You can find more description and usage examples at https://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git

How does it work

The svn2git repository gets you an application that will do the actual conversion. The conversion exists of looping over each and every commit in the subversion repository and matching the changes to a ruleset after which the changes are applied to a certain path in a git repo. The ruleset can specify which git repository to use and thus you can have more than one git repository as a result of running the conversion. Also noteworthy is that you can have a rule that, for example, changes in svnrepo/branches/foo/2.1/ will appear as a git-branch in a repository.

If you have a proper ruleset the tool will create the git repositories for you and show progress while converting commit by commit.

After it is done you likely want to run git repack -a -d -f to compress the pack file as it can get quite big.

Running as Docker image

Just mount your SVN folder, plus another working directory where Git repository will be created. Sample usage with input mounted in /tmp and output produced in /workdir:

docker build -t svn2git .
docker run --rm -it -v `pwd`/workdir:/workdir -v /var/lib/svn/project1:/tmp/svn -v `pwd`/conf:/tmp/conf svn2git /usr/local/svn2git/svn-all-fast-export --identity-map /tmp/conf/project1.authors --rules /tmp/conf/project1.rules --add-metadata --svn-branches --debug-rules --svn-ignore --empty-dirs /tmp/svn/

Building the tool

Run qmake && make. You get ./svn-all-fast-export. (Do a checkout of the repo .git' and run qmake and make. You can only build it after having installed libsvn-dev, and naturally Qt. Running the command will give you all the options you can pass to the tool.)

You will need to have some packages to compile it. For Ubuntu distros, use this command to install them all: sudo apt-get install build-essential subversion git qtchooser qt5-default libapr1 libapr1-dev libsvn-dev

To run all tests you can simply call the test.sh script in the root directory. This will run all Bats based tests found in .bats files in the directory test. Running the script will automatically execute qmake and make first to build the current code if necessary. If you want to run tests without running make, you can give --no-make as first parameter. If you want to only run a subset of the tests, you can specify the base-name of one or multiple .bats files to only run these tests like ./test.sh command-line svn-ignore. If you want to investigate the temporary files generated during a test run, you can set the environment variables BATSLIB_TEMP_PRESERVE=1 or BATSLIB_TEMP_PRESERVE_ON_FAILURE=1. So if for example some test in svn-ignore.bats failed, you can investigate the failed case like BATSLIB_TEMP_PRESERVE_ON_FAILURE=1 ./test.sh --no-make svn-ignore and then look in build/tmp to investigate the situation.

KDE

there is a repository kde-ruleset which has several example files and one file that should become the final ruleset for the whole of KDE called 'kde-rules-main'.

Write the Rules

You need to write a rules file that describes how to slice the Subversion history into Git repositories and branches. See https://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git. The rules are also documented in the 'samples' directory of the svn2git repository. Feel free to add more documentation here as well.

Rules

create respository

create repository REPOSITORY NAME
  [PARAMETERS...]
end repository

PARAMETERS is any number of:

  • repository TARGET REPOSITORY Creates a forwarding repository , which allows for redirecting to another repository, typically with some prefix.
  • prefix PREFIX prefixes each file with PREFIX, allowing for merging repositories.
  • description DESCRIPTION TEXT writes a DESCRIPTION TEXT to the description file in the repository

match

match REGEX
  [PARAMETERS...]
end match

Creates a rule that matches paths by REGEX and applies some PARAMETERS to them. Matching groups can be created, and the values used in the parameters. You need to make sure the regex matching a SVN directory path matches also the end slash (e.g. ./) otherwise Git fast-import will crash with an fatal: Empty path component found in input errors. For example, the rule /project/trunk/.*/myFolder, should become /project/trunk/.*/myFolder/.

PARAMETERS is any number of:

  • repository TARGET REPOSITORY determines the repository

  • branch BRANCH NAME determines which branch this path will be placed in. Can also be used to make lightweight tags with refs/tags/TAG NAME although note that tags in SVN are not always a single commit, and will not be created correctly unless they are a single copy from somewhere else, with no further changes. See also annotate true to make them annotated tags.

  • [min|max] revision REVISION NUMBER only match if revision is above/below the specified revision number

  • prefix PREFIX prefixes each file with PREFIX, allowing for merging repositories. Same as when used in a create repository stanza.

    • Note that this will create a separate commit for each prefix matched, even if they were in the same SVN revision.
  • substitute [repository|branch] s/PATTERN/REPLACEMENT/ performs a regex substitution on the repository or branch name. Useful when eliminating characters not supported in git branch names.

  • action ACTION determines the action to take, from the below three:

    • export I have no idea what this does
    • ignore ignores this path
    • recurse tells svn2git to ignore this path and continue searching its children.
  • annotated true creates annotated tags instead of lightweight tags. You can see the commit log with git tag -n.

include FILENAME

Include the contents of another rules file

declare VAR=VALUE

Define variables that can be referenced later. ${VAR} in any line will be replaced by VALUE.

Work flow

Please feel free to fill this section in.

Some SVN tricks

You can access your newly rsynced SVN repo with commands like svn ls file:///path/to/repo/trunk/KDE. A common issue is tracking when an item left playground for kdereview and then went from kdereview to its final destination. There is no straightforward way to do this. So the following command comes in handy: svn log -v file:///path/to/repo/kde-svn/kde/trunk/kdereview | grep /trunk/kdereview/mplayerthumbs -A 5 -B 5 This will print all commits relevant to the package you are trying to track. You can also pipe the above command to head or tail to see the first and last commit it was in that directory.

svn2git's People

Contributors

ad1217 avatar alef avatar alexgit2k avatar andersk avatar bekce avatar daelf avatar ditti4 avatar flokli avatar gabyx avatar harinath avatar hartwork avatar jmsantamaria avatar jobermayr avatar m-tmatma avatar mbaschnitzi avatar mclap avatar mgedmin avatar modax avatar neverpanic avatar nicolas17 avatar not-implemented avatar nsams avatar sdoerner avatar thiagomacieira avatar tnyblom avatar tobix avatar twisterrob avatar uqs avatar vampire avatar vterdunov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

svn2git's Issues

fast_import crashes but is not noticed

I created a script to do a simple reproduction: https://github.com/kfsone/examples/tree/master/svn-all-fast-import

But it actually proves to be pretty simple. Command line arguments:
--identity-map --identity-domain unknown.unk
--rules
--add-metadata
--stats
--empty-dirs
--propcheck
--svn-ignore
--svn-branches
--max-rev
file://${REPOS_PATH}

. Create svn repos,
. Create a commit with the svn path of: /repos/Trunk/test1.txt
. Create a rule to map /repos/Trunk to master
. Attempt to convert
-> warning about "/repos/" not matching
. Add rule to ignore /repos/ after /repos/Trunk rule
. Attempt to convert
-> conversion appears to complete
*> fast_import_crash_NNN created, git repository is empty

Warning for every new branch

I get this warning:
Exporting revision 1 .WARN: Branch "master" in repository "test1" doesn't exist at revision 1 -- did you resume from the wrong revision?

It seems this warning only appears when a new branch is created (for rev 1 there was no git created branch before). So I'm not sure if should care about this warning.

If this happens normally for every new branch I would feature request to not show this warning.

docker build broken

docker build -t svn2git . on master seems broken

g++ -m64 -Wl,-O1 -o ../svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o    -L/usr/lib/x86_64-linux-gnu -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -lQtCore -lpthread 
ruleparser.o: In function `QList<Rules::Repository::Branch>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x109): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Match::Substitution>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x1a1): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Repository>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x37b): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<Rules::Match>::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x4b8): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o: In function `QList<QList<Rules::Match> >::free(QListData::Data*) [clone .isra.0]':
ruleparser.cpp:(.text+0x599): undefined reference to `operator delete(void*, unsigned long)'
ruleparser.o:ruleparser.cpp:(.text+0xbd4): more undefined references to `operator delete(void*, unsigned long)' follow
collect2: error: ld returned 1 exit status
make[1]: *** [../svn-all-fast-export] Error 1
Makefile:110: recipe for target '../svn-all-fast-export' failed
make[1]: Leaving directory '/usr/local/svn2git/src'
Makefile:39: recipe for target 'sub-src-make_default' failed
make: *** [sub-src-make_default] Error 2
The command '/bin/sh -c cd /usr/local/svn2git && qmake && make' returned a non-zero code: 2

Import crash on big svn repo with "core git rejected index" error

I'm trying to migrate a huge svn repo (76G in size). It crashes in the middle after about 24 hours.

fast-import crash report:
    fast-import process: 15424
    parent process     : 23192
    at Tue Mar 20 09:41:13 2018

fatal: core git rejected index objects/pack/pack-2aa286062dcf4aa1d6bed58a2693a6cd00dba5a6.idx

Most Recent Commands Before Crash:

....

Active Branch LRU
-----------------
    active_branches = 1 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)   9999 refs/heads/master

Inactive Branches
-----------------
refs/heads/master:
  status      : active loaded
  tip commit  : de565fa294b19e927915f4343d192ed36aed7c2d
  old tree    : 0572758db74396179eb9473ab8e0fa881cf4cc33
  cur tree    : 0572758db74396179eb9473ab8e0fa881cf4cc33
  commit clock: 9999
  last pack   : 0

Not quite sure why the repo is that big tbh, cause it has only 19000 revisions. and we have other repo that has way more revisions but only a few G.

Assertion `svn_dirent_is_canonical(base, pool)' failed

Hi,

I am running the command below but it appears that there is an assertion failure. I tried with and without a leading / on the URI without any luck.

svn-all-fast-export --identity-map=authors.txt --rules=standard-rules-development.txt --stats --add-metadata https://svn.com/svn/MyRepo Invoked as:' svn-all-fast-export --identity-map=authors.txt --rules=standard-rules-development.txt --stats --add-metadata https://svn.com/svn/MyRepo' Loading rules from: "standard-rules-development.txt" Loading rules from "standard-rules-development.txt" Creating new repository "MyRepo" svn-all-fast-export: subversion/libsvn_subr/dirent_uri.c:972: svn_dirent_join: Assertion 'svn_dirent_is_canonical(base, pool)' failed. Aborted (core dumped)

Test name collision

I ran into this warning in CI logs:

bats warning: duplicate test name(s) in /home/runner/work/svn2git/svn2git/test/empty-dirs.bats: test_branching_with_svn-2dignore-2c_svn-2dbranches_and_empty-2ddirs_parameter_should_not_replace_filled_-2egitignore_files_with_empty_ones

I can confirm that we have two tests named branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones:

# git --no-pager grep -n 'should not replace'
test/empty-dirs.bats:1031:@test 'branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones' {
test/empty-dirs.bats:1063:@test 'branching with svn-ignore, svn-branches and empty-dirs parameter should not replace filled .gitignore files with empty ones' {

Both tests were added in fcb62af.

CC @Vampire

Build fails on Arch Linux

I can't get the build working, see the following

$ cat .qmake.stash 
QMAKE_CXX.QT_COMPILER_STDCXX = 201402L
QMAKE_CXX.QMAKE_GCC_MAJOR_VERSION = 8
QMAKE_CXX.QMAKE_GCC_MINOR_VERSION = 2
QMAKE_CXX.QMAKE_GCC_PATCH_VERSION = 1
QMAKE_CXX.COMPILER_MACROS = \
    QT_COMPILER_STDCXX \
    QMAKE_GCC_MAJOR_VERSION \
    QMAKE_GCC_MINOR_VERSION \
    QMAKE_GCC_PATCH_VERSION
QMAKE_CXX.INCDIRS = \
    /usr/include/c++/8.2.1 \
    /usr/include/c++/8.2.1/x86_64-pc-linux-gnu \
    /usr/include/c++/8.2.1/backward \
    /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include \
    /usr/local/include \
    /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed \
    /usr/include
QMAKE_CXX.LIBDIRS = \
    /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 \
    /usr/lib \
    /lib

make fails with the following:

[...] # snipped
g++ -Wl,-O1 -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -o ../svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o   -L -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -lQt5Core -lpthread   
/usr/bin/ld: svn.o: undefined reference to symbol 'svn_fs_file_contents'
/usr/bin/ld: /usr/lib/libsvn_fs-1.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:203: ../svn-all-fast-export] Error 1
make[1]: Leaving directory '/home/kartik/Projects/svn2git/src'
make: *** [Makefile:47: sub-src-make_first] Error 2

Identity-Map not working

Seems like the application is ignoring my identity-map file. I'm using the git-svn syntax in the file, and when i run svn-all-fast-export, it seems like it either thinks I'm not passing anything to that part or ignores it completely. any ideas?

Build errors

On Ubuntu 16.04 LTS:

Error:
$ qmake && make

cd src/ && ( test -e Makefile || /usr/lib/x86_64-linux-gnu/qt5/bin/qmake /home/me/src/svn2git/src/src.pro -o Makefile ) && make -f Makefile
make[1]: Entering directory '/home/me/src/svn2git/src'
g++ -c -m64 -pipe -O2 -Wall -W -D_REENTRANT -DVER=""5caa913b3ac6fea8ea3a92558889a12e7feed21e"" -DQT_NO_DEBUG -DQT_CORE_LIB -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4 -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I. -o ruleparser.o ruleparser.cpp
ruleparser.cpp:18:23: fatal error: QTextStream: No such file or directory
compilation terminated.
Makefile:211: recipe for target 'ruleparser.o' failed
make[1]: *** [ruleparser.o] Error 1
make[1]: Leaving directory '/home/me/src/svn2git/src'
Makefile:42: recipe for target 'sub-src-make_first' failed
make: *** [sub-src-make_first] Error 2

Dependencies are installed:
...with the exception that qt-qmake needed to be qt5-qmake (Documentation issue):

`$ sudo apt-get install qt-qmake' #this doesn't work

Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package qt-qmake

$ apt-cache policy build-essential subversion git qtchooser qt5-qmake qt5-default libapr1 libapr1-dev libsvn-dev

build-essential:
Installed: 12.1ubuntu2
Candidate: 12.1ubuntu2
Version table:
*** 12.1ubuntu2 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
subversion:
Installed: 1.9.3-2ubuntu1
Candidate: 1.9.3-2ubuntu1
Version table:
*** 1.9.3-2ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
git:
Installed: 1:2.7.4-0ubuntu1
Candidate: 1:2.7.4-0ubuntu1
Version table:
*** 1:2.7.4-0ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
qtchooser:
Installed: 52-gae5eeef-2build1gcc5.2
Candidate: 52-gae5eeef-2build1
gcc5.2
Version table:
*** 52-gae5eeef-2build1~gcc5.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
qt5-qmake:
Installed: 5.5.1+dfsg-16ubuntu7.2
Candidate: 5.5.1+dfsg-16ubuntu7.2
Version table:
*** 5.5.1+dfsg-16ubuntu7.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
100 /var/lib/dpkg/status
5.5.1+dfsg-16ubuntu7 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
qt5-default:
Installed: 5.5.1+dfsg-16ubuntu7.2
Candidate: 5.5.1+dfsg-16ubuntu7.2
Version table:
*** 5.5.1+dfsg-16ubuntu7.2 500
500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
100 /var/lib/dpkg/status
5.5.1+dfsg-16ubuntu7 500
500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
libapr1:
Installed: 1.5.2-3
Candidate: 1.5.2-3
Version table:
*** 1.5.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
libapr1-dev:
Installed: 1.5.2-3
Candidate: 1.5.2-3
Version table:
*** 1.5.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
libsvn-dev:
Installed: 1.9.3-2ubuntu1
Candidate: 1.9.3-2ubuntu1
Version table:
*** 1.9.3-2ubuntu1 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status

I do see:
$ ll /usr/lib/x86_64-linux-gnu/qt5/bin/qmake

-rwxr-xr-x 1 root root 3071144 Sep 19 2016 /usr/lib/x86_64-linux-gnu/qt5/bin/qmake*

Can you assist?

"create repository" ... doesn't

[When using dry-run]

As trivial as they come: first run, you sort of expect it to git init the directory for you if its absent, partly because of the amount of text to declare one:

create repository foo
end repository

but the absence of the repos still causes progress to halt, preventing you from doing a dry-run...

The use of the term "name" also sets up a false hope that the above would create a "foo.git" folder for me and use that.

Commits not visible after a "stuck" revision

I run svn-all-fast-export --rules standardlayout.rules --identity-map mapping.txt --svn-ignore --add-metadata --stats svn-repository
And it is processed up to latest revision 19018 according logs, and git bare repository size sounds correct.
But "master" reference is stuck to revision 10090, git log master last commit shown is svn path=/trunk/; revision=10090. Branches and tags created after that revision are not visible either.

If I run svn-all-fast-export again, it starts back to revision 10091... Investigating git bare repository, I find out as many objects/pack/tmp_pack_XXXX large files.

Do I guess right commits have not been "flushed" into repository ? What should I do to complete migration and get "recent" references visible ?

Thank you in advance for your help

Option to define a output subdirectory

I've successfully converted a large SVN repo.
Only use-case issue I'm facing is that all the matching SVN folders come together in the Git root folder.

SVN

--Product1/branches/1.0
----file1.txt
--Product2/branches/1.0
----file2.txt

Output to:

--/
--file1.txt
--file2.txt

I would like to specify a subfolder within the output repo, like:

match /([^/]+)/branches/([^/]+)/
  repository git-repo
  branch \2
  directory \1 -- statement example 
end match

Outputting as:

--Product1/file1.txt
--Product2/file2.txt

annoted tag can raise a fatal error

I don't have a clue why but for some SVN tags (aka branches), if you specify annotated true within a matching rule, the tool die with : 'fatal: Can't tag an empty branch.'
It doesn't happened without annotated.

I can't find anything with the failing tag that is different with those who works : could you lead to the path to understand what's happening ?

I'm using svn2git 1.0.11, svn 1.9.3 and git 2.15.1

1.0.14 fails to compile (while 1.0.13 compiles fine)?

Hi! I'm getting compile error with 1.0.14 that I'm not getting with 1.0.13. Any ideas what's wrong?

Environment

  • GCC 7.3.0
  • Subversion 1.11.1
  • Qt 5.11.1

Compiler invocation

x86_64-pc-linux-gnu-g++ -c -march=core2 -msse4.1 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=6144 -O2 -pipe -Wall -W -D_REENTRANT -fPIC -DVER="\"1.0.14\"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -isystem /usr/include/subversion-1 -isystem /usr/include/apr-1 -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -I. -I/usr/lib64/qt5/mkspecs/linux-g++ -o repository.o repository.cpp

Start of error output

repository.cpp: In function ‘QDataStream& operator<<(QDataStream&, const FastImportRepository::AnnotatedTag&)’:
repository.cpp:263:9: error: ambiguous overload for ‘operator<<’ (operand types are ‘QDataStream’ and ‘quint64 {aka long long unsigned int}’)
     out << annotatedTag.supportingRef
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         << annotatedTag.svnprefix
         ~~~~~~~~~~~~~~~~~~~~~~~~~
         << annotatedTag.author
         ~~~~~~~~~~~~~~~~~~~~~~
         << annotatedTag.log
         ~~~~~~~~~~~~~~~~~~~
         << (quint64) annotatedTag.dt
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

Full build log

(From trying to update the Gentoo package of it, hence the color and Gentoo bits)
svn2git-1.0.14-build-log.txt

[Feature Request] Support git `subtree` to reference a git repository from a prefix directory in another git repository

Currently i don't see the ability to use the git subtrees (better alternative to the git submodules) to represent a piece of directory in the GIT repo. Currently i can create multiple GIT repos from a single or several SVN repos, but can not make a reference repository in the GIT.

At least it was not mentioned here: https://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git

Do you have plans to make it?

Branches are missing from exported submodules due to missing export-and-recurse action

I am currently migrating a large SVN repository to Git which has a monolithic TTB structure (meaning: no subprojects, all branches are copies of the entire ^/trunk). During the conversion, most of the repository should go into a single large Git repository, except for one directory, which should move to a separate repository. I had some problems doing the conversion which I would like to describe first before asking some questions. To demonstrate my problem, I prepared a simplified toy example.

Attached below the issue you will find a bash script (show-issue) to reproduce the toy example from scratch.

The SVN repository

The SVN repository (example.svn) has two branches, the trunk (^/trunk) and a feature branch (^/branches/feature). Both branches have the same directory structure:

.
├── bar.txt
├── baz.txt
├── foo.txt
├── other
│   ├── bar.py
│   ├── baz.py
│   └── foo.py
└── src
    ├── bar.c
    ├── baz.c
    └── foo.c

As one can see from the following history, the feature branch was branched off the trunk in r4 and both branches received additional commits.

------------------------------------------------------------------------
r8 | msp | 2020-04-26 20:54:27 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   M /branches/feature/other/bar.py

Add more stuff to other files
------------------------------------------------------------------------
r7 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   M /branches/feature/bar.txt
   M /branches/feature/src/bar.c

Add more stuff to my files
------------------------------------------------------------------------
r6 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   M /trunk/other/bar.py

Add some stuff to other files
------------------------------------------------------------------------
r5 | msp | 2020-04-26 20:54:26 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   M /trunk/bar.txt
   M /trunk/src/bar.c

Add some stuff to my files
------------------------------------------------------------------------
r4 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   A /branches/feature (from /trunk:3)

Create feature branch
------------------------------------------------------------------------
r3 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   A /trunk/other
   A /trunk/other/bar.py
   A /trunk/other/baz.py
   A /trunk/other/foo.py

Add other files
------------------------------------------------------------------------
r2 | msp | 2020-04-26 20:54:25 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   A /trunk/bar.txt
   A /trunk/baz.txt
   A /trunk/foo.txt
   A /trunk/src
   A /trunk/src/bar.c
   A /trunk/src/baz.c
   A /trunk/src/foo.c

Add my files
------------------------------------------------------------------------
r1 | msp | 2020-04-26 20:54:24 +0200 (So, 26 Apr 2020) | 1 line
Changed paths:
   A /branches
   A /tags
   A /trunk

Create ttb structure
------------------------------------------------------------------------

First migration attempt (broken)

All files from the example.svn repository should go into a single large Git repository main.git, except for the contents of the other directory, which go to other.git. This here was my first attempt:

broken.rules

create repository broken/main.git
end repository

create repository broken/other.git
end repository


match /trunk/other/
  repository broken/other.git
  branch master
end match

match /trunk/
  repository broken/main.git
  branch master
end match

match /branches/([^/]+)/other/
  repository broken/other.git
  branch \1
end match

match /branches/([^/]+)/
  repository broken/main.git
  branch \1
end match

The Outcome

broken.log

+ svn-all-fast-export --debug-rules --add-metadata --rules broken.rules example.svn
WARNING; no identity-map or -domain specified, all commits will use default @localhost email address

Loading rules from: "broken.rules"
Loading rules from "broken.rules"
marksfile  "broken/main.git/marks-broken_main.git"
marksfile  "broken/other.git/marks-broken_other.git"
Invoked as:' svn-all-fast-export --debug-rules --add-metadata --rules broken.rules example.svn'

The content of the main.git repository looks ok:

##
## broken/main.git
##
* commit 1585deab140fde1c26e05caa2b3715c066523430
| Author: msp <msp@localhost>
| Date:   Sun Apr 26 18:54:26 2020 +0000
| 
|     Add more stuff to my files
|     
|     svn path=/branches/feature/; revision=7
| 
|  bar.txt   | 1 +
|  src/bar.c | 1 +
|  2 files changed, 2 insertions(+)
|   
| * commit faffe7f8e11b5a5077d79ede381250457e9e5014
|/  Author: msp <msp@localhost>
|   Date:   Sun Apr 26 18:54:26 2020 +0000
|   
|       Add some stuff to my files
|       
|       svn path=/trunk/; revision=5
|   
|    bar.txt   | 1 +
|    src/bar.c | 1 +
|    2 files changed, 2 insertions(+)
| 
* commit f8b88128f0b96a901ae7ca8abc16b3da36232df2
  Author: msp <msp@localhost>
  Date:   Sun Apr 26 18:54:25 2020 +0000
  
      Add my files
      
      svn path=/trunk/; revision=2
  
   bar.txt   | 1 +
   baz.txt   | 1 +
   foo.txt   | 1 +
   src/bar.c | 1 +
   src/baz.c | 1 +
   src/foo.c | 1 +
   6 files changed, 6 insertions(+)

tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.txt
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	src/foo.c

tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.txt
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	src/foo.c

But the feature branch of the other.git repository is 'detached' from the master branch and its tree is incomplete, i.e., it contains only the single file (bar.py) which was modified after branching (in r8).

##
## broken/other.git
##
* commit 0d7132382ede82f4df2c1fe0c1120b1c69fce221
  Author: msp <msp@localhost>
  Date:   Sun Apr 26 18:54:27 2020 +0000
  
      Add more stuff to other files
      
      svn path=/branches/feature/other/; revision=8
  
   bar.py | 2 ++
   1 file changed, 2 insertions(+)
  
* commit 87ff8db4f665a7a8637029b3d5cb5f4c777176e9
| Author: msp <msp@localhost>
| Date:   Sun Apr 26 18:54:26 2020 +0000
| 
|     Add some stuff to other files
|     
|     svn path=/trunk/other/; revision=6
| 
|  bar.py | 1 +
|  1 file changed, 1 insertion(+)
| 
* commit 604ca6dd5b379634d18e14f4d776e8d594a01075
  Author: msp <msp@localhost>
  Date:   Sun Apr 26 18:54:25 2020 +0000
  
      Add other files
      
      svn path=/trunk/other/; revision=3
  
   bar.py | 1 +
   baz.py | 1 +
   foo.py | 1 +
   3 files changed, 3 insertions(+)

tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.py

tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	bar.py

The Problem

I learned from UsingSvn2Git that the rule

match /branches/([^/]+)/other/
  repository broken/other.git
  branch \1
end match

won't be matched in r4 (A /branches/feature (from /trunk:3)), unless I define a recurse action

match /branches/([^/]+)/$
  action recurse
end match

However, I also need an export action for the main repository

match /branches/([^/]+)/
  repository broken/main.git
  branch \1
end match

and unfortunately svn-all-fast-export does not support an 'export-and-recurse' action.

Second migration attempt

The only solution I could come up with to fix that conflict was to create two separate rule files:

main.rules

create repository fixed/main.git
end repository

match /trunk/other/
  action ignore
end match

match /trunk/
  repository fixed/main.git
  branch master
end match

match /branches/([^/]+)/other/
  action ignore
end match

match /branches/([^/]+)/
  repository fixed/main.git
  branch \1
end match

other.rules

create repository fixed/other.git
end repository

match /trunk/other/
  repository fixed/other.git
  branch master
end match

match /branches/([^/]+)/$
  action recurse
end match

match /branches/([^/]+)/other/
  repository fixed/other.git
  branch \1
end match

match /
  action ignore
end match

MultiRules!!

Initially, the downside of this approach was that I had to call svn-all-fast-export twice, which doubled the conversion time. But then I discovered (from the source code) a handy feature, namely that you can specify a comma separated list of rules on the command line.

svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn

The two rule files are not merged, but instead executed independently in parallel. This feature was added in commit a741bdb by @tnyblom. This is an excellent feature IMO and unfortunately poorly documented.

The Outcome

This time, the result looks good:

fixed.log

+ svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn
WARNING; no identity-map or -domain specified, all commits will use default @localhost email address

Loading rules from: "main.rules"
Loading rules from "main.rules"
Loading rules from: "other.rules"
Loading rules from "other.rules"
marksfile  "fixed/main.git/marks-fixed_main.git"
marksfile  "fixed/other.git/marks-fixed_other.git"
Invoked as:' svn-all-fast-export --debug-rules --add-metadata --rules main.rules,other.rules example.svn

##
## fixed/main.git
##
* commit 1585deab140fde1c26e05caa2b3715c066523430
| Author: msp <msp@localhost>
| Date:   Sun Apr 26 18:54:26 2020 +0000
| 
|     Add more stuff to my files
|     
|     svn path=/branches/feature/; revision=7
| 
|  bar.txt   | 1 +
|  src/bar.c | 1 +
|  2 files changed, 2 insertions(+)
|   
| * commit faffe7f8e11b5a5077d79ede381250457e9e5014
|/  Author: msp <msp@localhost>
|   Date:   Sun Apr 26 18:54:26 2020 +0000
|   
|       Add some stuff to my files
|       
|       svn path=/trunk/; revision=5
|   
|    bar.txt   | 1 +
|    src/bar.c | 1 +
|    2 files changed, 2 insertions(+)
| 
* commit f8b88128f0b96a901ae7ca8abc16b3da36232df2
  Author: msp <msp@localhost>
  Date:   Sun Apr 26 18:54:25 2020 +0000
  
      Add my files
      
      svn path=/trunk/; revision=2
  
   bar.txt   | 1 +
   baz.txt   | 1 +
   foo.txt   | 1 +
   src/bar.c | 1 +
   src/baz.c | 1 +
   src/foo.c | 1 +
   6 files changed, 6 insertions(+)

tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.txt
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	src/foo.c

tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.txt
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.txt
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	src/bar.c
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	src/baz.c
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	src/foo.c

##
## fixed/other.git
##
* commit e76a6a71d849b1f1716d20e7132bb9433648da67
| Author: msp <msp@localhost>
| Date:   Sun Apr 26 18:54:27 2020 +0000
| 
|     Add more stuff to other files
|     
|     svn path=/branches/feature/other/; revision=8
| 
|  bar.py | 1 +
|  1 file changed, 1 insertion(+)
|   
| * commit 87ff8db4f665a7a8637029b3d5cb5f4c777176e9
|/  Author: msp <msp@localhost>
|   Date:   Sun Apr 26 18:54:26 2020 +0000
|   
|       Add some stuff to other files
|       
|       svn path=/trunk/other/; revision=6
|   
|    bar.py | 1 +
|    1 file changed, 1 insertion(+)
| 
* commit 604ca6dd5b379634d18e14f4d776e8d594a01075
  Author: msp <msp@localhost>
  Date:   Sun Apr 26 18:54:25 2020 +0000
  
      Add other files
      
      svn path=/trunk/other/; revision=3
  
   bar.py | 1 +
   baz.py | 1 +
   foo.py | 1 +
   3 files changed, 3 insertions(+)

tree of master branch:
100644 blob ce781c47e1cab657df5ca60339bf2a578defda8a	bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.py

tree of feature branch:
100644 blob f0f5c7c766cdb85df51efef2b02dff568788e0c0	bar.py
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca	baz.py
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	foo.py

Questions

Having to maintain two different rule files makes things a little bit complicated (in particular, since my real repository is not as simple as my example) and I was asking myself whether there was a different way to solve my problem using a single rules file, which I might have missed?

If not, would it be a great effort @tnyblom to add a new 'export-and-recurse' action and would this make it possible to have all rules in a single file (as in the broken attempt) using something like the following?

match /branches/([^/]+)/$
  repository broken/other.git
  branch \1
  action export-and-recurse
end match

Malformed rule shouldn't cause a core dump

Malformed rules shouldn't cause a core dump, it should just be an error, and the program should exit.

This was caused by a typo, in the rule set, instead of 'end match', I had 'emd match'

(gdb) bt
#0  0x00007f9fa3cbc1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f9fa3cbd8c8 in __GI_abort () at abort.c:90
#2  0x00007f9fa4af57b4 in qt_message_output (msgType=msgType@entry=QtFatalMsg, buf=<optimized out>) at global/qglobal.cpp:2323
#3  0x00007f9fa4af5959 in qt_message(enum QtMsgType, const char *, typedef __va_list_tag __va_list_tag *) (msgType=msgType@entry=QtFatalMsg,
    msg=msg@entry=0x4386a0 "Malformed line in rules file: line %d: %s", ap=ap@entry=0x7fff88320fb0) at global/qglobal.cpp:2369
#4  0x00007f9fa4af6261 in qFatal (msg=msg@entry=0x4386a0 "Malformed line in rules file: line %d: %s") at global/qglobal.cpp:2552
#5  0x000000000040d285 in Rules::load (this=0x1191aa0, filename=...) at ruleparser.cpp:317
#6  0x000000000040ff8b in load (this=<optimized out>) at ruleparser.cpp:124
#7  RulesList::load (this=this@entry=0x7fff88321b60) at ruleparser.cpp:39
#8  0x0000000000407209 in main (argc=6, argv=<optimized out>) at main.cpp:195
(gdb)

How to handle end-of-lines normalization

We are facing issues after migration when users add .gitattributes file to keep LF for shell scripts when working on Windows platform.
"Perfect world" expectation would be that svn-all-fast-export inspected svn:eol-style Subversion property and applies ".gitattributes" for each imported commit.

What is the recommended way to get all text files with LF EOL in repository itself, so that Git attributes can be used seamlessly after migration when committing from different EOL-style platforms?

Svn history not migrated for folders which were copied in Svn

I'm trying to convert a single SVN repo over to a single git repo. After migration of the svn repository to git I have noticed that git does not seem to follow svn copy operations so the resulting history is much briefer than I expect.

To illustrate - The existing history can be viewed in tortoise svn if you uncheck the "Stop on copy/rename" checkbox while viewing a folder's history but it cannot be viewed in Git after migration.

root

  • a

  • b

  • c

  • parent project

    • b

    • c

Projects b and c were recently copied under parent-proj as part of a restructuring effort with the intention of eventually deleting them from their old locations under root. After migration the resulting git repo is missing all of the history that originated from /b and /c before the move.

I have created a SO post for this here - https://stackoverflow.com/questions/53704582/retaining-svn-copy-history-when-converting-to-git-using-kde-svn2git-tool

Exporting revision 1 "/content/" did not match any rules; cannot continue

I am getting the error: Exporting revision 1 "/content/" did not match any rules; cannot continue

/content/ is a directory in trunk and each branch, but why would it be part of the match? I'm really confused.

My SVN rules are:

#
# Declare the repositories we know about:
#

create repository pw_content
end repository

#
# Declare the rules
# Note: rules must end in a slash
#

match /trunk/
  repository pw_content
  branch master
end match

match (branch|tag)/([^/]+)/
 repository pw_content
  branch \2
end match

std::bad_alloc on large commit

I'm trying to import a very large and ugly SVN repo on the order of 22GB. In this commit the following are deleted:

new/
     dir1
     dir2
     dir3

And the following are copied from points in the history

new/
     subdir/
               dir1
               dir2
               dir4

After running for almost 4 hours I get the following error:
terminate called after throwing an instance of 'bad_alloc'
what(): std::bad_alloc

the fast_import_crash_xxx says:
fata: EOF in data (xxxx bytes remaining)

Build on OSX?

Any help getting this to build on osx?

Thanks to brew, I think I have almost everything installed, but I'm getting the error below. I think I'm missing some svn libs, but I'm not certain how to get them. Hoping someone else has done this before and can help.

Thanks!

qmake && make
Info: creating stash file /Users/macfly/dmjohns/svn-all-fast-export/.qmake.stash
cd src/ && ( test -e Makefile || /usr/local/opt/qt/bin/qmake -o Makefile /Users/macfly/dmjohns/svn-all-fast-export/src/src.pro ) && /Library/Developer/CommandLineTools/usr/bin/make -f Makefile
Project WARNING: Qt has only been tested with version 10.13 of the platform SDK, you're using 10.14.
Project WARNING: This is an unsupported configuration. You may experience build issues, and by using
Project WARNING: the 10.14 SDK you are opting in to new features that Qt has not been prepared for.
Project WARNING: E.g., 10.14 enables dark mode and layer-backed views, which Qt 5.11 does not support.
Project WARNING: Please downgrade the SDK you use to build your app to version 10.13, or configure
Project WARNING: with CONFIG+=sdk_no_version_check when running qmake to silence this warning.
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o ruleparser.o ruleparser.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o repository.o repository.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o svn.o svn.cpp
svn.cpp:528:13: warning: 'svn_fs_paths_changed2' is deprecated [-Wdeprecated-declarations]
SVN_ERR(svn_fs_paths_changed2(&changes, fs_root, pool));
^
/usr/local/include/subversion-1/svn_fs.h:1754:1: note: 'svn_fs_paths_changed2' has been explicitly marked deprecated here
SVN_DEPRECATED
^
/usr/local/include/subversion-1/svn_types.h:56:42: note: expanded from macro 'SVN_DEPRECATED'

define SVN_DEPRECATED attribute((deprecated))

                                     ^

svn.cpp:569:13: warning: 'svn_fs_revision_proplist' is deprecated [-Wdeprecated-declarations]
SVN_ERR(svn_fs_revision_proplist(&revprops, fs, revnum, pool));
^
/usr/local/include/subversion-1/svn_fs.h:2854:1: note: 'svn_fs_revision_proplist' has been explicitly marked deprecated here
SVN_DEPRECATED
^
/usr/local/include/subversion-1/svn_types.h:56:42: note: expanded from macro 'SVN_DEPRECATED'

define SVN_DEPRECATED attribute((deprecated))

                                     ^

2 warnings generated.
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o main.o main.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DVER=""83d6f8d2b28e7db5c1ac959df96fc74af363ab92"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -I/usr/include/subversion-1 -I/usr/local/include/subversion-1 -I/usr/include/apr-1.0 -I/usr/include/apr-1 -I/usr/local/include/apr-1 -I/usr/local/Cellar/qt/5.11.2/lib/QtCore.framework/Headers -I. -I/usr/local/Cellar/qt/5.11.2/mkspecs/macx-clang -F/usr/local/Cellar/qt/5.11.2/lib -o CommandLineParser.o CommandLineParser.cpp
/Library/Developer/CommandLineTools/usr/bin/clang++ -stdlib=libc++ -headerpad_max_install_names -arch x86_64 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.11 -Wl,-rpath,@executable_path/Frameworks -o ../svn-all-fast-export.app/Contents/MacOS/svn-all-fast-export ruleparser.o repository.o svn.o main.o CommandLineParser.o -F/usr/local/Cellar/qt/5.11.2/lib -lsvn_fs-1 -lsvn_repos-1 -lapr-1 -lsvn_subr-1 -framework QtCore -framework DiskArbitration -framework IOKit
ld: library not found for -lsvn_fs-1
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [../svn-all-fast-export.app/Contents/MacOS/svn-all-fast-export] Error 1
make: *** [sub-src-make_first] Error 2

CI? GitHub Actions? Travis?

Hi!

Are there any plans to add CI here? Would you welcome PRs with basic GitHub Actions or Travis integration?

Best, Sebastian

freezing at first revision with content

I have to migrate an old svn repository to git.
I tried if my rules are working with --dry-run until all looked good, but when i'm starting the tool without the dry-run parameter it freezes at the first revision with content

Exporting revision 7487 nothing to do
Exporting revision 7488 nothing to do
Exporting revision 7489 .

At that spot it's frozen for hours without any relevant cpu or memory usage
After a while i tried to start it with --debug-rules and could see that it founds a match for first file in the first revision with content and then does nothing for ... some time

Exporting revision 7487 nothing to do
Exporting revision 7488 nothing to do
Exporting revision 7489 rev 7489 /TMS/trunk/tms/.classpath matched rule: "../svn2git.rules:4 /TMS/trunk/" exporting.
.add/change file ( /TMS/trunk/tms/.classpath -> "master" "tms/.classpath"`

  • I installed cygwin and am running it on a win10 pc
  • I made a local copy of our svn repository with 'svnrdump dump' and 'svnadmin load'

I call

~/svn2git/svn-all-fast-export.exe --debug-rules --identity-map ../trunk/authors-transform.txt --rules ../svn2git.rules local

and my current rules are pretty simple

create repository tms
end repository

match /TMS/trunk/
	repository tms
	branch master
end match

match /TMS/branches/Feature_Branches/([^/]+)/
	repository tms
	branch feature/\1
end match

match /TMS/branches/([^/]+)/
	repository tms
	branch \1
end match

match /TMS/tags/([^/]+)/
	repository tms
	branch ref/tags/\1
	annotated true
end match

Missing docs

Documentation is missing for: annotate, substitute repository and substitute branch, and decoding what they do is a little hard.

Rules in the form /project/tags/([^/]+)/dir_name/ are not being fetched into the new git-repository

The structure of my svn repository looks similar to this:

/Poject/
    trunk/
        dir1/
        dir2/
        dir3/
         ...
         dir<n>/
    branches/
        branch_1/
            dir1/           
            dir2/           
            dir3/    
        branch_2/
            dir1/           
            dir2/           
            dir3/        
            ...
            dir<n>/ 
    tags/
        tag_1/
            dir1/           
            dir2/           
            dir3/    
        tag_2/
            dir1/           
            dir2/           
            dir3/     
            ...      
            dir<n>/

In reality there are several branches and several tags. Most of the tags were generated after the branches and the name of these tags is the same as the corresponding branch. The repository has over 26000 revisions.

The svn-repository is splitted into several git repositories (in total 8) where some of the directories (dir1, ..., dir<n>) were grouped together, for example:

  • git_repo_1 = dir1 + dir4 +dir5
  • git_repo_2 = dir2
  • git_repo_3 = dir6 + dir10
    ...

Here is the rules file for this repository:

# Declare the repositories we know about:
#

create repository git_repo_1 
end repository
create repository git_repo_2
end repository
create repository git_repo_3
end repository


#
# git_repo_1 
#
# git_repo_1 = dir1 + dir4 +dir5
#

match /Project/trunk/(dir1|dir4|dir5)/
  repository git_repo_1 
  prefix \1/
  branch master
end match
match /Project/branches/([^/]+)/(dir1|dir4|dir5)/
  repository git_repo_1 
  prefix \2/
  branch \1
end match
match /Project/tags/([^/]+)/(dir1|dir4|dir5)/
  repository git_repo_1 
  prefix \2/
  branch refs/tags/v\1
  annotated true
end match

#
# git_repo_2 = dir2
#

match /Project/trunk/dir2/
  repository git_repo_2
  branch master
end match
match /Project/branches/([^/]+)/dir2/
  repository git_repo_2 
  branch \1
end match
match /Project/tags/([^/]+)/dir2/
  repository git_repo_2
  branch refs/tags/v\1
  annotated true
end match


#
# git_repo_3
#
# git_repo_3= dir6 + dir10
#

match /Project/trunk/(dir6 |dir10)/
  repository git_repo_3
  prefix \1/
  branch master
end match
match /Project/branches/([^/]+)/(dir6 |dir10)/
  repository git_repo_3
  prefix \2/
  branch \1
end match
match /Project/tags/([^/]+)/(dir6 |dir10)/
  repository git_repo_3
  prefix \2/
  branch refs/tags/v\1
  annotated true
end match

match /
end match

Results:

  • After performing the migration not all the tags were fetched into the corresponding git-repositories.

  • In the logs there was found a message the following message for some repositories: recursiveDumpDir: "/Project/tags/tag_1/dir2/" skip entry for different/ignored repository

Questions:

Why are the tags not fetched?

Why did the message recursiveDumpDir appear in the logs even though none of the match definitions contains the parameter action export and all them have defined the parameter repository?

Branch name doesn't conform to GIT standards

I receive the following error while attempting to run a migration - "fatal: Branch name doesn't conform to GIT standards: refs/tags/AbsoluteSafe 2.0 (v301)"
Any suggestions on what can be done about this - I'm not sure if this is a tool bug, or a limitation of the characters used in the name of the tag/branch.

Handling of empty directories

Svn2git knows the "--empty-dirs" option to handle empty SVN directories. Unfortunately the option is only effective if the directory is created empty. If a directory is emptied while deleting all files and folders in it, the .gitignore file is not created.
It means that currently - if you have such cases - some kind of 'post-processing' after the svn2git-based migration is needed.

gitlab

A lot of people are leaving github expecting that the microsoft take over will cause the site to become a mess in not too much time. As has happened with other such examples (like Skype).

I reserved the groups 'svn2git' and 'svn-all-fast-export' on gitlab, so hit me up if the current maintainers are interested in migrating and using one (or both) of those groups.

Resuming import deletes commits made to git repository not from the import itself

I am trying to export a Subversion repository that contains multiple releases of a product into a new git repository. Along the way I am also export code from a different system (Serena PVCS) by hand into this same repository. My goal has been to do the following:

  • Use svn2git to export revisions 1-1000 (for example) into the new git repostory, which loads "release 1" of our product into git.
  • Manually commit the code for "release 1" of our other product from PVCS into git.
  • Use svn2git to export revisions 1001-2000 into the repository, which loads "release 2" of our product into git.
  • Manually commit the code for "release 2" from PVCS.

I do the manual commits by cloning the bare repository svn2git created into a new folder. I check out master, add all my code from PVCS, commit, and push back into the bare repository.

However, when I run svn2git the second time to load the next batch of revisions, it actually completely wipes out my manual commits in the bare repository, as if they had never existed. They don't show up in the git log within the bare repository. When I "git pull" in my local copy I created for adding the code from PVCS, it reports divergence of the remote branch and does a commit for a merge.

Each time I run svn2git it seems to wipe out my commits that didn't come from subversion.

Am I doing the process incorrectly, or is this just not possible, to run svn2git into what is essentially an existing repository that has code from other commits?

Please improve documentation of incremental conversions

First of all, thank you for sharing this very flexible and highly configurable conversion tool.

I had however some nasty problems getting incremental conversions working, which I'd like to share because they were partially caused by a lack of documentation and partially by a design flaw:

Description of the problems

The --resume-from flag

The --resume-from flag is poorly documented. The svn2git --help output simply states

       --resume-from         start importing at svn revision number
       --max-rev             stop importing at svn revision number

which misguided me to think that the flags were meant to be used for incremental conversions. That is, if the previous svn2git run converted all revisions up to revision <n>, I thought that I needed to call svn2git --resume-from <n+1> ... for incrementally converting the newer revisions.

In the end, it turned out that the --resume-from isn't needed at all, but the necessary information is collected automatically from the log files (see below). The --resume-from flag seems to be meant to be used for discarding and rewriting some commits which were already committed. (I did not really check that in the source thoroughly.)

The importance of the log files

The log files (log-<repo>.git) play a crucial role for the incremental conversions. Unfortunately, this important detail is completely undocumented. What makes things worse is the fact that the log-<repo>.git is stored outside the Git repository, contrary to other important auxiliary files (branchNotes-<repo>.git, marks-<repo>.git).

.
├── <repo>.git
│   ├── HEAD
│   ├── branchNotes-<repo>.git
│   ├── branches
│   ├── config
│   ├── description
│   ├── hooks
│   ├── info
│   ├── marks-<repo>.git
│   ├── objects
│   └── refs
└── log-<repo>.git

Since the term "log file" suggests that the file just contains a transcript from a previous run, I failed to include them when I moved the converted Git repositories from my own machine to the production server (to run incremental conversions triggered by an SVN post-commit hook). The first run of the incremental update produced tons of warnings

Exporting revision <nnnn> ..WARN: Branch "master" in repository "<repo>.git" doesn't exist at revision <nnnn> -- did you resume from the wrong revision?

and left the Git repositories in a broken state. (A lot of files were missing from HEAD^{tree}).

Conclusion

In view of my experiences, I would like to suggest the following enhancements:

  • Improve the documentation of the --resume-from and --max-rev options, in particular how they relate to incremental conversions.
  • Improve the documentation of the auxiliary files (log-<repo>.git, branchNotes-.git, marks-.git`) and how they relate to incremental conversions.
  • Move the log-<repo>.git into the Git repository (in a backward-compatible way).

Remaining old files if SVN copy operations replace directories

I had to migrate a large SVN repo but run into trouble.
After analyzing the problem a bit, it seems that SVN copy operations for directories which replace
older directories are problematic, because svn2git does not delete all the old content in this case
(as SVN does it).

Are there any options to work around (or resolve) the problem? Will it be possible to fix this in the tool?

To illustrate the problem, I created a very small test repository showing the problem. The output of "svn log -v" can be found in attached file svn_log.txt.

Basically a file in trunk is created (folder_a/file1.txt), then a branch is created (branch1) from trunk,
the first file is deleted and another file is created (folder_a/file2.txt) and another branch from trunk is created (branch2).
Finally the directory folder_1 from branch2 is copied to branch1 replacing the older directory folder_a.
In SVN the older file folder_a/file1.txt is then (implicitly) deleted. But after migrating to Git it is
still there :-(

svn_log.txt

The command '/bin/sh -c cd /usr/local/svn2git && qmake && make' returned a non-zero code: 1

Step 4/7 : ADD . /usr/local/svn2git
---> Using cache
---> 8b89fdb5ba3b
Step 5/7 : RUN cd /usr/local/svn2git && qmake && make
---> Running in 7b3a74125f6e
Usage: /usr/lib/x86_64-linux-gnu/qt4/bin/qmake [mode] [options] [files]

QMake has two modes, one mode for generating project files based on
some heuristics, and the other for generating makefiles. Normally you
shouldn't need to specify a mode, as makefile generation is the default
mode for qmake, but you may use this to test qmake on an existing project

Mode:
-project Put qmake into project file generation mode
In this mode qmake interprets files as files to
be built,
defaults to *.c; *.ui; *.y; *.l; *.ts; *.xlf; *.qrc; *.h; *.hpp; *.hh; *.hxx; *.H; *.cpp; *.cc; *.cxx; *.C
Note: The created .pro file probably will
need to be edited. For example add the QT variable to
specify what modules are required.

Warnings Options:
-Wnone Turn off all warnings; specific ones may be re-enabled by
later -W options

The command '/bin/sh -c cd /usr/local/svn2git && qmake && make' returned a non-zero code: 1

svn:external

Hello,

I want to migrate old SVN to git

the SVN repository is like:

http://svn/Sky/trunk/folder1/
http://svn/Sky/trunk/folder2/
http://svn/Sky/branches/0.1/folder1/
http://svn/Sky/branches/0.1/folder2/

...

http://svn/Sky/branches/X.Y.Z/folder1/
http://svn/Sky/branches/X.Y.Z/folder2/

but folder2 is an svn:external from http://svn/Moon/tags/18_10_2013/sources/folder2 project at a specific tag on trunk.

and

folder2 is an svn:external from http://svn/Moon/tags/01_10_2013/sources/folder2 project at a specific tag on /branches/0.1.

and so... for /branches/X.Y.Z/

How can i write the match rules in those cases?

BR

Crash on exit: pointer being freed was not allocated

svn-all-fast-export (the latest version of the code in this repository as of 2015-12-01), installed using the svn2git port I recently added to MacPorts, crashes after doing a conversion, on OS X 10.11 El Capitan, built using Qt 5.5.1, Subversion 1.9.3 and APR 1.5.2 also installed by MacPorts.

The terminal output ends with:

svn-all-fast-export(14669,0x7fff71d09000) malloc: *** error for object 0x7fc2ca80d400: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

The crash log OS X writes says:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
abort() called
*** error for object 0x7fc2ca80d400: pointer being freed was not allocated


Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x00007fff8a619002 __pthread_kill + 10
1   libsystem_pthread.dylib             0x00007fff856ff5c5 pthread_kill + 90
2   libsystem_c.dylib                   0x00007fff8efa46e7 abort + 129
3   libsystem_malloc.dylib              0x00007fff93ab0070 free + 425
4   libapr-1.0.dylib                    0x0000000109b071b3 apr_allocator_destroy + 27
5   libapr-1.0.dylib                    0x0000000109b07761 apr_pool_terminate + 36
6   libsystem_c.dylib                   0x00007fff8efa546b __cxa_finalize_ranges + 345
7   libsystem_c.dylib                   0x00007fff8efa576f exit + 55
8   svn-all-fast-export                 0x0000000109a4b89b start + 59

I don't know if this is relevant, but this Subversion mailing list post says crashes in apr_pool_terminate could be due to failing to clear an error with svn_error_clear.

Assertion failed

~/folder$ svn-all-fast-export --identity-map=../../authors.txt --rules=../rules.txt --stats --add-metadata https://svn.repo.com/folder-with-trunk-branches-tags
Invoked as:' svn-all-fast-export --identity-map=../../authors.txt --rules=../rules.txt --stats --add-metadata https://svn.repo.com/folder-with-trunk-branches-tags'
Loading rules from: "../rules.txt" 
Loading rules from "../rules.txt" 
Creating new repository "my-new-git-repo" 
svn-all-fast-export: /build/subversion-yttRtj/subversion-1.9.3/subversion/libsvn_subr/dirent_uri.c:972: svn_dirent_join: Assertion `svn_dirent_is_canonical(base, pool)' failed.
Aborted (core dumped)

../rules.txt < https://gist.github.com/Vampire/f48bd7e3667a0cc0cbe1b5746c492be8

../../authors.txt < list of user = Name <[email protected]>

Is it possible to handle moving subpathes

I'm evaluating how to migrate our svn-repositories to git. At the moment we have multi project repositories. So we need to export subpathes to different git repositories. The problem is that the structure was changed over the time.

At the moment I try the following approach:

  1. run log -xml -v <mySubPath>
  2. find in which revisionRange has a specific path
  3. make an specific migration for this range.

Let explain it on a simple example with the following history

  • r1 Directory a1 was created
  • r5 Director a1 was renamed to a5
  • r10 Directory a5 was rename to b

I'm unsure if this is possible with svn-all-fast-export. Can someone give me a hint?

Handling of merges that contained empty directories

Hello,

we have the following situation:

  • a commit on an SVN branch introduced two empty directories.
  • this branch is later merged on the trunk

Applying the svn2git tool with the option --empty-dirs, the commit on the SVN branch added, as expected, two .gitignore files.

However, the subsequent merge commits did not carry this information, and I sort-of see the point: only the content of the SVN merge commit got transcribed and reproduced, but it creates this weird situation where the master (ex trunk) does not contain exactly the same content at the other branch, despite having done a merge.

Would there be any work-around on this? The easy solution is to reperform the merge, at the end of the migration, where this time the empty .gitignore will be picked, but if there would be a cleaner way, that would be nice.

Thanks in advance, and thank you for the tool!

Failed to write to process: Error writing to process | fatal: Empty path component found in input

Hi, I'm trying to convert an old svn repo and I'm getting this problem.

The crash report:

fast-import crash report:
    fast-import process: 15204
    parent process     : 1
    at 2019-08-20 09:28:46 +0000

fatal: Empty path component found in input

Most Recent Commands Before Crash
---------------------------------
  data 2224
  blob
  mark :18446744073709551171
  data 5330
  blob
  mark :18446744073709551170
  data 11902
  blob
  mark :18446744073709551169
  data 2693
  blob
  mark :18446744073709551168
  data 1629
  blob
  mark :18446744073709551167
  data 1733
  blob
  mark :18446744073709551166
  data 2569
  blob
  mark :18446744073709551165
  data 8825
  blob
  mark :18446744073709551164
  data 1803
  blob
  mark :18446744073709551163
  data 3467
  blob
  mark :18446744073709551162
  data 3363
  blob
  mark :18446744073709551161
  data 7116
  blob
  mark :18446744073709551160
  data 4363
  blob
  mark :18446744073709551159
  data 3269
  blob
  mark :18446744073709551158
  data 3212
  blob
  mark :18446744073709551157
  data 16104
  blob
  mark :18446744073709551156
  data 5243
  blob
  mark :18446744073709551155
  data 6632
  blob
  mark :18446744073709551154
  data 23172
  blob
  mark :18446744073709551153
  data 10553
  blob
  mark :18446744073709551152
  data 7209
  blob
  mark :18446744073709551151
  data 1092
  blob
  mark :18446744073709551150
  data 4020
  blob
  mark :18446744073709551149
  data 3713
  blob
  mark :18446744073709551148
  data 4967
  blob
  mark :18446744073709551147
  data 3157
  blob
  mark :18446744073709551146
  data 3856
  blob
  mark :18446744073709551145
  data 5849
  blob
  mark :18446744073709551144
  data 4291
  blob
  mark :18446744073709551143
  data 3751
  blob
  mark :18446744073709551142
  data 7776
  blob
  mark :18446744073709551141
  data 25184
  commit refs/heads/master
  mark :19
  committer user <user@localhost> 1190996168 +0000
  data 39
  deleteall
* M 100644 :18446744073709551614 /.gitignore

Active Branch LRU
-----------------
    active_branches = 1 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)      0 refs/heads/master

Inactive Branches
-----------------
refs/heads/master:
  status      : active loaded dirty
  tip commit  : 9e09d0745d5af0c3978f948e10fd135c07783625
  old tree    : 0000000000000000000000000000000000000000
  cur tree    : 0000000000000000000000000000000000000000
  commit clock: 0
  last pack   :


Marks
-----
  exported to marks-REPO

-------------------
END OF CRASH REPORT

svn-ignore switch apparently has no effect

Hi there,

I'm currently working on migrating a huge SVN repo into Git using your tool. I found it to do the job quite nicely (thanks for that!), however I have stumbled across one slight issue:

I'm passing the following parameters: --identity-map xyz.authors --rules xyz.rules --add-metadata --stats --svn-ignore --commit-interval 1000 /tmp/svn/

However, in the resulting repo, I see no gitignore files although svn ignore properties are existing in the repo.

Am I doing something wrong or is there some way to further troubleshoot this problem?

Thanks in advance and kind regards,
Philipp

Build on Windows?

It may be just my ignorance, but even after installing the whole QT (which takes longer than Visual Studio!) I still cannot build it on my Windows machine. Anyone who publishes Windows build instructions and/or makes it more user friendly will have my blessing!

1 Svn Repository to 60 Git Repositories - Out of memory, malloc failed

Hello. I'm trying to use svn-all-fast-export to migrate our SVN project.

Out team used SVN Repository very not useful

they made a main repository, and put sub repositories to that main repository
so main repository is over 30000 revisions, and have lots of file size ...

Main Repository
|- Sub Repository 1
| |-branches
| |- tags
| |- trunk
|- Sub Repository 2
| |-branches
| |-tags
| |-trunk
....

By the way, I need to separate these sub repository to each git repositories.
(Sub)Repository 1
|-branches
|-tags
|-trunk
(Sub)Repository 2
|-branches
|-tags
|-trunk
...

I typed these things to Rule file, and svn2git program worked.
but, when passing 3000 revisions, Out of memory error happened.

I thought it using too much git repositories, I tried to convert just one repository to rule files,
but it rejected because of rule is not correct.
because, program may read other sub repository, because i didn't typed to rule file about it.

Error message is

fatal : Out of memory, malloc failed (tried to allocate 1468122 bytes)
warning: die() called many times. Recursion error or racy threaded death!
fatal: internal consistency error creating the index

is there any good way to solve this problem..? T_T

svn:global-ignores not considered when generating .gitignore files and patterns are not transformed properly

@alexgit2k implemented the ability to generate .gitignore entries from svn:ignore properties.
"Unfortunately" since SVN 1.8 there is also svn:global-ignores which basically is the same, but works on all subdirectories, not only on the directory where it is specified, like /*.java vs. *.java in .gitignore.

It would be great if svn2git would also support the svn:global-ignores which it currently only warns about if you ask it to. @alexgit2k could you maybe give it a shot? :-)

Segmentation fault processing svn repository

I get a segmentation fault while processing a SVN repository. There is no information other than:

fatal: EOF in data (285 bytes remaining)

It creates a dump report but I don't know what the contents means. It prints '+' to the screen over and over and it never appears to move past that point. It's like it has encountered a circular dependency in the svn repository.

Here's the back trace from the core dump:

[New LWP 26037]
[New LWP 26038]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `svn2git/svn-all-fast-export --identity-map ../tools/authors/authors.txt --rules'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007faaad763de1 in QByteArray::append (this=this@entry=0x189ef78, ba=...) at tools/qbytearray.cpp:1616
1616            d->data[d->size] = '\0';
(gdb) bt
#0  0x00007faaad763de1 in QByteArray::append (this=this@entry=0x189ef78, ba=...) at tools/qbytearray.cpp:1616
#1  0x0000000000418c61 in append (s=..., this=0x189ef78) at /usr/include/QtCore/qstring.h:997
#2  FastImportRepository::Transaction::addFile (this=0x189ef30, path=..., mode=33261, length=236) at repository.cpp:899
#3  0x000000000042259e in dumpBlob (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=<optimized out>, finalPathName=..., pool=pool@entry=0x7faa99e92038) at svn.cpp:292
#4  0x0000000000423fff in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdd2218) at svn.cpp:340
#5  0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xda01d8) at svn.cpp:335
#6  0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd8f1c8) at svn.cpp:335
#7  0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdc6208) at svn.cpp:335
#8  0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd6b188) at svn.cpp:335
#9  0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdbb1f8) at svn.cpp:335
#10 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xdb11e8) at svn.cpp:335
#11 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=fs_root@entry=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xd861b8) at svn.cpp:335
#12 0x0000000000423f53 in recursiveDumpDir (txn=txn@entry=0x189ef30, fs_root=0xd3f6f0, pathname=..., finalPathName=..., pool=pool@entry=0xcf81f8) at svn.cpp:335
#13 0x0000000000426e07 in SvnRevision::exportInternal (this=this@entry=0x7fff44b63ca0, key=key@entry=0x189d8d8 "/FusionWorks/branches/FW_5_5_0_226", change=change@entry=0xcf9780,
    path_from=<optimized out>, path_from@entry=0xd4ee87 "/branches", rev_from=rev_from@entry=149644, current=..., rule=..., matchRules=...) at svn.cpp:914
#14 0x000000000042b56b in SvnRevision::exportDispatch (this=this@entry=0x7fff44b63ca0, key=key@entry=0x189d8d8 "/branches/FW_5_5_0_226", change=change@entry=0xcf9780,
    path_from=0xd4ee87 "/FusionWorks/branches", rev_from=149644, changes=changes@entry=0xcf96b0, current=..., rule=..., matchRules=..., pool=pool@entry=0xd4ee08) at svn.cpp:716
#15 0x000000000042cff0 in SvnRevision::exportEntry (this=this@entry=0x7fff44b63ca0, key=0x189d8d8 "/FusionWorks/branches/FW_5_5_0_226", change=0xcf9780, changes=0xcf96b0) at svn.cpp:666
#16 0x000000000042f091 in SvnRevision::prepareTransactions (this=this@entry=0x7fff44b63ca0) at svn.cpp:536
#17 0x000000000042f660 in SvnPrivate::exportRevision (this=<optimized out>, revnum=revnum@entry=149645) at svn.cpp:444
#18 0x000000000042f7ec in Svn::exportRevision (this=this@entry=0x7fff44b63e40, revnum=revnum@entry=149645) at svn.cpp:154
#19 0x0000000000407754 in main (argc=6, argv=<optimized out>) at main.cpp:276
(gdb)

Unable to use --resume-from

I'm trying to perform what is admittedly a rather complex svn2git conversion, as documented here:

https://sourceforge.net/p/brlcad/code/HEAD/tree/brlcad/trunk/misc/svn2git/

When I try to do a partial conversion and resume it with the --resume-from option, the process gets a few commits in and then crashes. It generates a log, and this seems to be the key issue:

fatal: mark :0 not declared

looking at the mark file in the repo, it does indeed start with :1 with no :0 entry present. Is this a known issue?

1.0.15 fails to compile (different error than 1.0.14)

Hi! I'm getting compile error with 1.0.15 that I'm not getting with 1.0.13. Any ideas what's wrong?

Environment

  • GCC 8.2.0
  • Subversion 1.12.0
  • Qt 5.12.3

Compiler invocation

x86_64-pc-linux-gnu-g++ -c -march=core2 -msse4.1 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=6144 -O2 -pipe -Wall -W -D_REENTRANT -fPIC -DVER="\"1.0.15\"" -DQT_NO_DEBUG -DQT_CORE_LIB -I. -I. -isystem /usr/include/subversion-1 -isystem /usr/include/apr-1 -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -I. -I/usr/lib64/qt5/mkspecs/linux-g++ -o repository.o repository.cpp

Start of error output

repository.cpp: In member function ‘virtual void FastImportRepository::restoreBranchNotes()’:
repository.cpp:471:34: error: variable ‘QDataStream branchNotesStream’ has initializer but incomplete type
     QDataStream branchNotesStream(&branchNotesFile);
                                  ^
repository.cpp: In member function ‘virtual void FastImportRepository::saveBranchNotes()’:
repository.cpp:798:34: error: variable ‘QDataStream branchNotesStream’ has initializer but incomplete type
     QDataStream branchNotesStream(&branchNotesFile);
                                  ^

Full build log

(From trying to update the Gentoo package of it, hence the color and Gentoo bits)
svn2git-1.0.15-build-log.txt

Related

Can you tag a new release?

Title says it all... Easy for package distributors to provide a new version, as the current one is from 2013.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.