hartwork/svneverever

:flashlight: Collects path entries across SVN history (Python)

About

svneverever helps you inspect the structure of a SVN repository and the changes made to it over time. Its most common use is in combination with svn-all-fast-export (or "KDE's svn2git" if you wish).

Installation

Dependencies

svneverever requires Python 3.8 or higher and PySVN (the one on SourceForge, not the one on PyPI). If you want to install svneverever with pip you need a few additional packages. For Debian/Ubuntu the following should work:

# sudo apt install python3-svn python3-pip

System package manager

Some distributions offer a native package for svneverever:

• Arch AUR
• Funtoo
• Gentoo
• openSUSE

pip

Install with pip as user to avoid messes with Python system files.

# pip install --user svneverever

From source

# git clone https://github.com/hartwork/svneverever.git
# cd svneverever
# python3 setup.py install --user
# echo 'export PATH="${HOME}/.local/bin:${PATH}"' >> ~/.bashrc

Usage

svneverever needs the "server-side" of the repository with full history. There are two ways to obtain this. Let's take the SVN history of headphone effect library bs2b as an example.

The first way is to directly point svneverever to the server svneverever svn://svn.code.sf.net/p/bs2b/code/. However this is slow and it puts a lot of stress on the server, and will be little fun to run multiple times.

The second and recommended method is first downloading the history with svnrdump dump (that comes with Subversion >=1.7) or rsvndump (with a slightly different feature set and supporting older versions of Subversion). For this method we first need to use svnadmin to create an empty repository and then load the output of svnrdump/rsvndump into it. This can be done in the following way:

With svnrdump dump

# svnadmin create bs2b_svn_store

# time sh -c 'svnrdump dump svn://svn.code.sf.net/p/bs2b/code/ | svnadmin load bs2b_svn_store/'
[..]
real    0m3.008s
user    0m0.309s
sys     0m0.235s

With rsvndump

# svnadmin create bs2b_svn_store

# time sh -c 'rsvndump svn://svn.code.sf.net/p/bs2b/code/ | svnadmin load bs2b_svn_store/'
[..]
real    2m54.403s
user    0m1.003s
sys     0m1.300s

The output can now be obtained by pointing svneverever to the svn store directory bs2b_svn_store.

# svneverever --no-progress bs2b_svn_store/

Analyzing the output

The output of the direct method or the rsvndump method will be the same and looks like this:

Analyzing 175 revisions...

(  1; 175)  /branches
( 66;  76)      /libbs2b-config-header
                    [..]
(  1; 175)  /tags
(109; 175)      /description
                    [..]
( 46; 175)      /libbs2b
                    [..]
( 28;  46)      /libbs2b-2.2.1
                    [..]
( 31; 175)      /plugins
                    [..]
(  1; 175)  /trunk
( 80; 175)      /description
( 80; 175)          /img
(  2; 175)      /libbs2b
(  2; 175)          /doc
(  2;  80)              /img
(  2;   6)              /src
(  4; 175)          /m4
(  2; 175)          /src
(  2; 175)          /win32
(  2; 175)              /bs2bconvert
(  2; 175)              /bs2bstream
(  2; 175)              /examples
(  2; 175)              /sndfile
(  2; 175)      /plugins
( 38; 175)          /audacious
( 38; 175)              /m4
( 38; 175)              /src
( 24; 175)          /foobar2000
(143; 175)          /ladspa
(144; 175)              /m4
(143; 175)              /src
( 39; 175)          /qmmp
(  2; 175)          /vst
(  2; 175)              /src
(  2; 175)                  /resources
(  2; 175)              /win32
(117; 175)          /winamp
(  2; 175)          /xmms
( 12; 175)              /m4
( 12; 175)              /src

The ranges on the left indicate at what revision folders appeared first and got deleted latest.

You can see a few things in this output:

• That a branch called libbs2b-config-header got deleted at revision 76.
• In which order plug-ins in plugins appeared over time.
• That tag libbs2b-2.2.1 was deleted at the same revision that a tag folder libbs2b appeared.

The last item we can verify to see if it was actually moved into that tag subfolder.

# svneverever --no-progress --tags --flatten bs2b_svn_store/ | grep '2.2.1$'
Analyzing 175 revisions...

(110; 175)  /tags/description/2.2.1
( 47; 175)  /tags/libbs2b/2.2.1
( 28;  46)  /tags/libbs2b-2.2.1

Next I have a look at who the committers where, when they joined or left and how many commits the did (though that last number is of limited value). This can help to write an identity map for svn2git.

# svneverever --no-progress --committers bs2b_svn_store/
Analyzing 175 revisions...

 81 (  1; 174)  boris_mikhaylov
 94 (  4; 175)  hartwork

--help output

# svneverever --help
usage: svneverever [-h] [--version] [--committers] [--no-numbers]
                   [--no-progress] [--non-interactive] [--tags] [--branches]
                   [--no-dots] [--depth DEPTH] [--flatten]
                   [--unknown-committer NAME]
                   REPOSITORY

Collects path entries across SVN history

positional arguments:
  REPOSITORY            Path or URI to SVN repository

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

mode selection arguments:
  --committers          Collect committer names instead of path information
                        (default: disabled)

common arguments:
  --no-numbers          Hide numbers, e.g. revision ranges (default: disabled)
  --no-progress         Hide progress bar (default: disabled)
  --non-interactive     Will not ask for input (e.g. login credentials) if
                        required (default: ask if required)

path tree mode arguments:
  --tags                Show content of tag folders (default: disabled)
  --branches            Show content of branch folders (default: disabled)
  --no-dots             Hide "[..]" omission marker (default: disabled)
  --depth DEPTH         Maximum depth to print (starting at 1)
  --flatten             Flatten tree (default: disabled)

committer mode arguments:
  --unknown-committer NAME
                        Committer name to use for commits without a proper
                        svn:author property (default: "<unknown>")

Please report bugs at https://github.com/hartwork/svneverever.  Thank you!

svneverever's Issues

Diff target 'http://<PATH>' was not found in the repository at revisions '0' and '1'

Hi !
Just try svneverever because I need to migrate a SVN repo to git.
The one repo consists of several projects in different pathes.

When I try to checkout a specific path I got this error:

Analyzing 39397 revisions...
  0.00%  (00:00:07 remaining)  [                                                                                                                                                                                               ]Traceback (most recent call last):
  File "/home/user/.local/bin/svneverever", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.10/site-packages/svneverever/__main__.py", line 370, in main
    summary = client.diff_summarize(
pysvn._pysvn.ClientError: Diff target 'http://svnserver:82/svn/ZW/CAN2x' was not found in the repository at revisions '0' and '1'

Any chance to fix this?

svneverever --committers fails on commits without committer information

$ python2 svneverever --committers $MYREPO
Analyzing 7078 revisions...
100.00%  (00:00:00 remaining)  [###################################################################################################]

Traceback (most recent call last):
  File "svneverever", line 17, in <module>
    main()
  File "modules/svneverever/main.py", line 321, in main
    dump_nick_stats(nick_stats, digit_count(latest_revision), config=args)
  File "modules/svneverever/main.py", line 111, in dump_nick_stats
    print(format % (commit_count, first_commit_rev, last_commit_rev, _encode(nick)))
  File "modules/svneverever/main.py", line 61, in _encode
    return text.encode(sys.stdout.encoding or 'UTF-8', 'replace')
AttributeError: 'NoneType' object has no attribute 'encode'

ModuleNotFoundError: No module named 'svneverever.main'

I am trying to use svneverever. When I install, it fails when I try to run svneverever with an error:
ModuleNotFoundError: No module named 'svneverever.main'

I've created a github repo duplicating this error.

https://github.com/yakton/svneverever-in-a-container
The requirements to run these examples in both cases are docker and docker-compose.

Installing svneverever using pip

docker-compose run svneverever-pip
And then running the command svneverever

root@8910fed2c381:/# svneverever
Traceback (most recent call last):
  File "/usr/local/bin/svneverever", line 6, in <module>
    from svneverever.main import main
ModuleNotFoundError: No module named 'svneverever.main'
root@8910fed2c381:/#

Installing svneverever using git

docker-compose run svneverever-git
And then running the command svneverever

root@5c00c5fa7e50:/svneverever# svneverever
Traceback (most recent call last):  File "/root/.local/bin/svneverever", line 11, in <module>
    load_entry_point('svneverever==1.4.1', 'console_scripts', 'svneverever')()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'svneverever.main'

Thanks in advance.

Use of "nonlocal" breaks support of Python 2

Probably, Python 2 support needs to be dropped altogether.

Unable to open repository

I tried the following commands

svn co https://plugins.svn.wordpress.org/schreikasten
svneverever schreikasten

i get

ERROR: Unable to connect to a repository at URL 'file:///home/flip111/php/tmp/schreikasten'
Unable to open repository 'file:///home/flip111/php/tmp/schreikasten'

What are the usage instructions? Should i follow the example literally and create a dummy repository?

go from svneverever output to svn2git input

I've read the blog post here https://blog.hartwork.org/?p=763 there it says:

With that information at hand it's easy to write a svn2git rules file for it

I'm totally new to this. I don't know where to start on this step. Are there any pointers?

Not resolvable python dependencies on Windows

For the python 3.7.x at least these:

• fcntl
• termios

The first one can be workarounded with a bit of hack:
https://stackoverflow.com/a/25471508/2672125

When the second one seems just not resolvable at all:
https://stackoverflow.com/questions/52062977/unable-to-import-termios-package

So, i don't see a way to use it from Windows.

AttributeError: 'Client' object has no attribute 'info2'

Traceback (most recent call last):
  File "C:\Users\ME\AppData\Roaming\Python\Python38\site-packages\svnevere
ver\__main__.py", line 312, in main
    latest_revision = client.info2(
AttributeError: 'Client' object has no attribute 'info2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python38\lib\runpy.py", line 194, in _run_module_as_mai
n
    return _run_code(code, main_globals, None,
  File "c:\program files\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\ME\AppData\Roaming\Python\Python38\Scripts\svneverever.ex
e\__main__.py", line 7, in <module>
  File "C:\Users\ME\AppData\Roaming\Python\Python38\site-packages\svnevere
ver\__main__.py", line 314, in main
    except (pysvn.ClientError) as e:
AttributeError: module 'pysvn' has no attribute 'ClientError'