ionos-cloud/reprepro

Debian package repository producer

reprepro with multiple versions support

This git repository hosts a branch for reprepro which adds multiple versions support to it. See Debian bug #570623 for details and updates.

The upstream repository can be found on https://salsa.debian.org/brlink/reprepro

Release Notes

The multiple-versions patch set adds following features:

• Add shunit2 based tests (Closes: #857302)
• Support multiple versions. (Closes: #570623)
• Add the commands move, movesrc, movematched, movefilter
• Add Limit and Archive option

How to keep multiple versions

The default behavior of this reprepro is identical to upstream's version. To keep multiple versions of the same package in the archive, you have to set the Limit option to the desired maximum amount (or to 0 for unlimited). See the description in the man page for details.

Database layout changes

The database layout changes from the upstream release to the multiple versions patch set. The difference is as following:

upstream

• packages.db maps "package name" to "control file" without duplicates
• no packagenames.db

multiple versions

• packages.db maps "package name|version" to "control file" without duplicates
• packagenames.db maps "package name" to "package name|version" allowing duplicates and duplicates sorted by dpkg --compare-versions descending

The first time the database is opened by reprepro with multiple versions support, the database will be upgraded from the upstream layout to the multiple versions layout. Warning: There is no way back (but could be done with a simple Python script)!

Howto rebase

1. Rebase the multiple-versions branch on top of the updated upstream master branch and push it to https://salsa.debian.org/bdrung/reprepro/

2. Refresh the multiple-versions-debian branch by taking the upstream debian branch. Apply patch debian: Switch to dh and Run shunit2 tests on build time. Cherry-pick all commits from multiple-versions. Then apply patch Add trace debugging output, debian: Update changelog and Add README.md describing this git branch.

reprepro's Issues

Please add information about the multi-version changes

Could you write a README-multiple-versions file which explains the theory of these additions, what's needed (upgrade path etc) and how to use it?

Including two versions of the same package with the same .orig.tar.* fails and mucks up references.db

If I try to includedsc two source packages into the same component and distribution, and these two source packages include the original source, and the original source is the same for both (like if the debian version changed), the following error occurs:

BDB0696 Duplicate data items are not supported with sorted data
Internal error of the underlying BerkeleyDB database:
Within references.db subtable references at put: BDB0067 DB_KEYEXIST: Key/data pair already exists

Once it reaches this point, it has already added entries for the dsc and debian.tar.* files to resources.db. These changes are not reversed upon encountering the error, so the state of the repo is inconsistent.

[Feature Request] Version in tag

First off, thanks for offering this "multiple-versions" feature to reprepro.
Right now, the tag does not reflect the version available in debian/changelog:

# git tag --list
5.1.1-multiple-versions-2017-02-10
5.1.1-multiple-versions-2017-02-27
5.1.1-multiple-versions-2017-03-27
5.1.1-multiple-versions-2017-03-28
5.1.1-multiple-versions-2017-03-29
5.1.1-multiple-versions-2017-04-12

Changing the tag format to reflect debian/changelog version would help us automate package building, for instance with a format like v5.1.1-2.1.
The branch name is explicit.

ddeb packages don't show up when querying the repository with apt/apt-get

I've switched to using this fork of reprepro for my repos for its ability to have multiple versions and support for .ddeb files.

I've created repos and I think I've successfully imported the .ddeb packages. The command reprepro -T ddeb list bionic shows them, and they're there in the repository if you browse the filesystem tree of the repository using a web browser, but when accessing the repository from a remote machine using apt or apt-get it can't find any of the -dbgsym packages.

The Components/DDebComponents lines in my conf/distributions file are:

Components: main
DDebComponents: main

And my sources.list on the remote machine is just has the standard:

deb <url> bionic main

If I'm doing something wrong either with importing the .ddeb files into the repo, or how my sources.list for the repo is configured, I don't know what it is.

The only thing I can think of is that maybe it has something to do with the .ddeb packages not showing up in the main/binary-amd64/Packages file but instead in the main/debug/binary-amd64/Packages file. But I'm not sure what to do on the sources.list side to get apt to see them.

Thanks.

Database migration screws up database names

packages.db from my instance of vanilla 5.1.1 contains databases with the following names:

bionic|main|amd64
bionic|main|i386
bionic|main|source
bionic|pkgbuild|amd64
bionic|pkgbuild|i386
bionic|pkgbuild|source
u|bionic|main|amd64
u|bionic|main|i386
u|bionic|pkgbuild|amd64
u|bionic|pkgbuild|i386
u|xenial|main|amd64
u|xenial|main|i386
u|xenial|pkgbuild|amd64
u|xenial|pkgbuild|i386
xenial|main|amd64
xenial|main|i386
xenial|main|source
xenial|pkgbuild|amd64
xenial|pkgbuild|i386
xenial|pkgbuild|source

(presented in the order in which they appear in a dump from db_dump)

After multiversion 5.1.1 migrates packages.db, it contains databases with the following names:

bionic|main|amd64
bionic|main|i3864
bionic|main|source
bionic|pkgbuild|amd64
bionic|pkgbuild|i3864
bionic|pkgbuild|source
d|bionic|main|amd64rce
d|bionic|main|i3864rce
d|bionic|pkgbuild|amd64
d|bionic|pkgbuild|i3864
d|xenial|main|amd643864
d|xenial|main|i38643864
d|xenial|pkgbuild|amd64
d|xenial|pkgbuild|i3864
u|bionic|main|amd643864
u|bionic|main|i38643864
u|bionic|pkgbuild|amd64
u|bionic|pkgbuild|i3864
u|xenial|main|amd643864
u|xenial|main|i38643864
u|xenial|pkgbuild|amd64
u|xenial|pkgbuild|i3864
xenial|main|amd64|i3864
xenial|main|i3864|i3864
xenial|main|sourcei3864
xenial|pkgbuild|amd6464
xenial|pkgbuild|i386464
xenial|pkgbuild|source4

Additionally, any data that was contained in databases whose names have changed is not migrated.

I believe the culprit is line 2107 of database.c. Instead of using Key.data directly, it should probably be copied using strndup with Key.size, as is done in lines 372-377. It will have to be freed each time, however. Perhaps a strncpy+realloc solution would be more elegant.

Is there a way to "Limit: x" different components of a distribution differently?

latest dpkg-deb zstd compression is not supported by reprepro

Ubuntu impish
reprepro: 5.3.0-1.1 multiple-versions (630bb1e)
dpkg: 1.20.9ubuntu2

Ubuntu has switched default dpkg-deb compression from xz to zstd: cf. changelog.
Adding a deb package built using the latest dpkg-deb to a repository managed by reprepro now leads to:

reprepro --component stable -Vb . includedeb impish ../../linux-buildinfo-5.13.0-12-generic_5.13.0-12.12_amd64.deb
Could not find a suitable control.tar file within '../../linux-buildinfo-5.13.0-12-generic_5.13.0-12.12_amd64.deb'!
There have been errors!

There is no such issue when adding packages built with dpkg 1.20.9.
Unfortunately, downgrading dpkg to the latter prevents any further installation of binary packages built and distributed by Ubuntu.

possible upstream fix

Looks like upstream has changed in debian experimental according to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570623

Successfully built on bookworm and it looks like it's working for me.

On update: Internal error of the underlying BerkeleyDB database

I just updated Debian from bullseye to bookworm and shortly after that (having called the bookworm version of reprepro successfully from my unchanged scripts) I updated just the reprepro package to the Debian experimental version (5.4.2-1). Immediately my scripts triggered the below error during calls to reprepro includedeb. But after a few repetitions of the same command (I tried to bring up a reproducible case) I suddenly found that the error had stopped from appearing (I did not downgrade in the meantime, reprepro: This is reprepro version 5.4.2).

My gut feeling is that we're looking at a "database" update that crashes in between a "transaction" which no ACID properties, thus after each crash the "old" record is gone, but the "new" record never landed in the "database".

BDB0696 Duplicate data items are not supported with sorted data
Internal error of the underlying BerkeleyDB database:
Within references.db subtable references at put: BDB0067 DB_KEYEXIST: Key/data pair already exists

re-adding distribution fails

Hi,

I am using this reprepro branch to mange multiple distributions.
It's working great, beside a bug that I found when adding an distribution that has been deleted before.

I did the following steps:

• deleted an distribution from conf/distributions
• run reprepro --delete clearvanished.
• added same distribution again
• run reprepro export or pull

Then I get this message:

$ reprepro pull trusty-test
Calculating packages to pull...
/var/repo//db/packages.db/trusty-test|main|source: BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
Internal error of the underlying BerkeleyDB database:
Within packages.db subtable trusty-test|main|source at c_get(DB_NEXT): BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
There have been errors!

Using db_dump, I found out, that all entries about "trusty-test" are deleted from packages.db, but not from packages.secondary.db.
I assume that only the rows of the removed database must be deleted from packages.secondary.db and it will work.

Best regards

Achim

"reprepro update" does not take multiple versions of one package, but the latest only

Is that by design or it's a bug?

Require build-depend on db-util

Trying to build 5.1.1-multiple-versions gives me:

./basic.sh
test_empty
I: Calling /usr/src/tmp/reprepro-profitbricks/reprepro -b ./testrepo list buster
./basic.sh: 28: /usr/bin/shunit2: db_verify: not found
ASSERT:BerkeleyDB 'packages.db' is broken.
[etc]

Adding db-util to the Build-Depends line in debian/control helped.

Repo dies and DB files get corrupted when switching from 5.3.0 non-multi to multiple-versions-debian

The problem

Users start complaining so you go check that and see that your repo is half-dead, dist files became really small, for example: dists/buster/main/binary-amd64/Packages may become just a few kilobytes of size when it has to be much bigger in big repos. People stop seeing packages in the repo's "index", but deb files are there still.

reprepro check returns:

packages.db/stretch|main|armhf: BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
Internal error of the underlying BerkeleyDB database:
Within packages.db subtable stretch|main|armhf at c_get(DB_NEXT): BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
There have been errors!

The cause

When you update reprepro binary to a newer version and then execute includedeb after that - dist files start containing only what you have just added. After that you may run reprepro check and start seeing the error message above.
The thing is that originally there is no db/packagenames.db file and it is created on first run, but it's somehow already broken.

To avoid that issue

Hold your breath, don't add anything YET ... Once reprepro binary is updated simply do reprepro check. After that do whatever you want.

If already broken!

Don't worry, it seems that you don't have to have a backup this time!
To FIX everything, simply:

• remove broken db/packagenames.db
• run reprepro check to populate db/packagenames.db in a right way
• run reprepro export to re-generate those almost-empty dist files so they are good and healthy and heavy.
• next time it's not gonna break, keep using reprepro.

Few issues/concerns

Hi,

This is awesome thank you. I'm finding that using the ls or list command gives me a "segmentation fault" is there a way I can send you a log of sorts ?

I'm also finding by adding the multi feature it's not very intuitive to remove Packages, I think it would be much easier to have a command line option which you can specify if you want to replace the package if it already exists in the repo or not. Or if reprepro supported search queries like

remove somepackage (<<3.0)

Rather than having to specify each version to remove.

I think having a commandline option like --multi, allows multiple versions of the same package in a repo, if the option is not used old versions of the package would be removed when a new is imported.

Thanks

Upgrade existing repository

Hi,

Will this work with an existing reprepro install? Kinda convert the existing db to supported structure?

remove a specific version of a package

This fork of reprepro works fine for me, but sometimes I need to remove a specific version of a package from a component but there is no such option.
I have multiple version of a package and those packages can have multiple status (stable, updates, releases), If I made a mistake and change the component of a package there is no option to rollback in this case just remove the package.

Thanks.

Not pulling some packages

I have a reprepro setup like so:

# conf/distributions 
Origin: Mendel
Label: Mendel
Codename: eagle
Architectures: arm64
Components: main
Description: Mendel Linux Mirror
Update: - mendel
Log: logfile

# conf/updates
Name: mendel
Method: https://mendel-linux.org/apt/eagle
FilterList: hold list
VerifyRelease: blindtrust

And my list file contains lots of things including:

libavfilter7:arm64				install
libavformat-dev:arm64				install
libavformat58:arm64				install

...

libusb-1.0-0:arm64                              install
libusb-1.0-0-dev:arm64                          install

When I run reprepro -b . --noskipold -v --show-percent update eagle I see it pull lots and lots of other packages specified in my conf/list file but for some reason it completely skips libusb-1.0-0 and libavfilter, libavformat, etc. In fact, if you look at pool the liba and libu directories are entirely missing:

$ ls pool/main/
a  b  c  d  e  f  g  h  i  j  k  l  libc  libd  libe  libh  libi  libj  libl  libs  libt  libu  libw  libx  m  n  o  p  r  s  t  u  v  w  x  z

I don't get any error messages and the files are available in the source repo (e.g. curl http://mendel-linux.org/apt/eagle/pool/main/libu/libusb-1.0/libusb-1.0-0_1.0.22-2_arm64.deb works as expected).

Any ideas?

INTERNAL ERROR: Package database is not sorted!!!

reprepro multiple-versions-debian branch.
I started seeing The lock file './db/lockfile' already exists. There might be another instance with the error same database dir running. To avoid locking overhead, only one process but there is no other reprepro running at the same time.
This is what I found in the logs. Note
reprepro: archallflood.c:310: save_package_version: Assertion `false' failed.

build	16-Mar-2021 14:55:30	Exporting indices...
error	16-Mar-2021 14:55:30	INTERNAL ERROR: Package database is not sorted!!!
error	16-Mar-2021 14:55:30	reprepro: archallflood.c:310: save_package_version: Assertion `false' failed.
error	16-Mar-2021 14:55:30	Aborted
error	16-Mar-2021 14:55:30	The lock file './db/lockfile' already exists. There might be another instance with the
error	16-Mar-2021 14:55:30	same database dir running. To avoid locking overhead, only one process
error	16-Mar-2021 14:55:30	can access the database at the same time. Do not delete the lock file unless
error	16-Mar-2021 14:55:30	you are sure no other version is still running!
error	16-Mar-2021 14:55:30	There have been errors!
error	16-Mar-2021 14:55:30	The lock file './db/lockfile' already exists. There might be another instance with the
error	16-Mar-2021 14:55:30	same database dir running. To avoid locking overhead, only one process
error	16-Mar-2021 14:55:30	can access the database at the same time. Do not delete the lock file unless
error	16-Mar-2021 14:55:30	you are sure no other version is still running!
error	16-Mar-2021 14:55:30	There have been errors!

https://files.freeswitch.org/repo/deb/debian-unstable/conf/distributions
https://files.freeswitch.org/repo/deb/debian-unstable/db/

Simple command to forget/unreference missing files?

I'd like to remove files from the repo massively with my file manager. And then just tell reprepro to remove references to those that were missing. Can I do that?

[Feature Request] Supersedes packages by source package name

Imagine how Ubuntu PPA works:

I can have a source package named MyApp-1.0 building MyApp_1.0.0.deb, and MyApp-1.1 building MyApp_1.1.0.deb. I want the user use the latest one by default, but keep the old version there just in case something in the new version is broken and I can't fix it in time in the new version, so it's always an option for rolling-back.

Now I want to backport something to 1.0.0, it will be 1.0.1 or 1.0.0-1, whatever, I want it supersedes 1.0.0.deb but does not touch 1.1.0.deb. On Ubuntu PPA, this is quite simple: I just release the 1.0.1 or 1.0.0-1 package with the same source package name MyApp-1.0 rather than MyApp-1.0.1.

So if I use the official reprepro, I can't have multiple versions, even if they come from different source packages.
If I use this fork, I have to manually remove the superseded versions.

I would suggest make use of negative Limit value.

• 1 means keeping only one version of a binary package
• -1 can mean keeping only one version of a binary package from each source package.

"File is already registered with different checksums!" during "reprepro update"

"reprepro update" does not success with:

File "pool/unstable/b/blah/blah_1.1.0~232~0ac2e9c.orig.tar.xz" is already registered with different checksums!
[...]
There have been errors!

Is there a way to ignore that?

Error "Package database is not sorted!!!" by update command

Hello,

we are just having a look at using reprepro for setting up several internal repositories for example for Icinga2. As we also would like to have older package versions in these repositories I played a bit around with reprepro.

After installation and configuration I was only able to download the latest version from (in this case) http://packages.icinga.com/debian/pool/main/i/icinga2/. I wasn't able to get the older versions for Debian Stretch too.

If I try to download them manually und put them into the repository by using the option includedeb everything seems to work correctly. I'm able to have a look at all versions with the command ls icinga2:

icinga2 | 2.8.0-1.stretch | icinga-stretch | i386, amd64
icinga2 | 2.7.2-1.stretch | icinga-stretch | i386, amd64
icinga2 | 2.7.1-2.stretch | icinga-stretch | i386, amd64
icinga2 | 2.7.1-1.stretch | icinga-stretch | i386, amd64
icinga2 | 2.7.0-1.stretch | icinga-stretch | i386, amd64
icinga2 | 2.6.3-1.stretch | icinga-stretch | i386, amd64

But If I use the command reprepro --noskipold update I just get the following error message:

Calculating packages to get...
Package database is not sorted!!!
reprepro: upgradelist.c:135: save_package_version: Assertion `false' failed.
Aborted

Is there a possibility to get this working as expected or am I only able to get the latest version from a repository and newer ones after this while using reprepro? Or do I just have to use reprepro update without option --noskipold?

Error adding ddeb packages

I'm using current 5.1.1-multiple-versions branch in Ubuntu Bionic.
When I'm adding package without *.ddeb there is no error:

$ reprepro -b ~/z/repo/ubuntu/bionic -T 'dsc|deb' -C main include bionic reprepro_5.1.1-2.2+bionic_amd64.changes 
Exporting indices...

If I'm trying to add package with *.ddeb an error occurs:

$ reprepro -b ~/z/repo/ubuntu/bionic -T 'dsc|deb|ddeb' -C main include bionic reprepro_5.1.1-2.2+bionic_amd64.changes 
Could not find 'main' in components of 'bionic': '
Deleting files just added to the pool but not used.
(to avoid use --keepunusednewfiles next time)
There have been errors!

My conf/distributions:

Origin: Ubuntu
Label: Ubuntu
Suite: stable
Codename: bionic
Version: 18.04
Architectures: amd64 i386 source
Components: main contrib
UDebComponents: main contrib

How to migrate from 5.1.1 to 5.1.1 + multiple?

It said that "The multiple-versions reprepro supports migrating from 5.1.1 to 5.1.1 + multiple versions " in the readme. But I can't find a way to make it in the document.
Can anyone point me out?
Thanks very much!