aptly-dev/aptly

aptly - Debian repository management tool

aptly's Issues

aptly publish multiple component from multiple mono component snapshots

Andrey said:

"What I see is that snapshot should be mono-component (it should contain packages from one component only). So most probably mirror should be mono-component. But when publishing aptly should give an option to publish several mono-component snapshots under one prefix/distribution so that published repository is multi-component once again."

Latest version when merging 3+ snapshots?

Let's say I mirror the following and update daily:
* http://us.archive.ubuntu.com/ubuntu/ precise main
* http://us.archive.ubuntu.com/ubuntu/ precise-updates main
* http://us.archive.ubuntu.com/ubuntu/ precise-security main

After I update each mirror I create a snapshot for the day:
* snapshot precise-04202014
foo_1.0.1.deb
bar_1.0.1.deb
baz_1.0.1.deb
* snapshot precise-updates-04202014
foo_1.0.3.deb
bar_1.0.2.deb
* snapshot precise-security-04202014
foo_1.0.2.deb
bar_1.0.3.deb

Now I would like to create a combined snapshot with the latest package from precise-04202014 & precise-updates-04202014 & precise-security-04202014:
* snapshot precise-all-04202014
foo_1.0.3.deb
bar_1.0.3.deb
baz_1.0.1.deb

Is there a way I can do this?

With "aptly snapshot merge" I do not get the latest across multiple snapshots because "packages with the same name-architecture pair are replaced during merge (package from latest snapshot on the list wins)".

aptly.info sample config file is missing a line terminator

"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false
"dependencyFollowAllVariants": false,

The sample config block in the documentation fails due to a missing comma after dependencyFollowRecommends

Atomic switching between published snapshots

There seem to be a small hole when switching from publishing one snapshot to another on the same url. Ex:

aptly publish snapshot -distribution=dev-testing dev-testing-20130314-140332
... add new package to dev-testing and make a new snapshot ...
aptly publish drop dev-testing
aptly publish snapshot -distribution=dev-testing dev-testing-20130314-145115

Between the last two commands there is a small period of time where the dev-testing repository doesn't exist. If a server happen to be checking the repository at that time, it will fail.

While I have only browsed the source cursory, it does seem it would be possible to implement a publish swtich-snapshot command, that I believe should roughly consist of:

1. Add any packages added by the new snapshot
2. Update repository meta-data
3. Remove existing packages not referenced in the new snapshot.

aptly mirror update ERROR: unable to update: download errors

Hi there,

by updating my repo mirror the following error occured:
Downloading http://ftp.ch.debian.org/debian/pool/main/liba/libalgorithm-diff-perl/libalgorithm-diff-perl_1.19.02-2.dsc...
ERROR: unable to update: download errors:
unexpected EOF
unexpected EOF

when I restarted the operation, there was nothing in the download queue to be downloaded. How can I be sure the mirror update was successful?

Best Regards

Better documentation of the config file

Hello Andrey! Let me say that aptly is great! I'm migrating from reprepro to aptly and everything is smooth. Thanks!

I have a problem with some parameters like architectures, where doc says "architectures is a list of architectures to process; if left empty defaults to all available architectures; could be overridden with option -architectures"

I left it empty, but when I tried to publish a snapshot with this command line:

# aptly publish snapshot -distribution="quantal" 20140318-stable
Loading packages...

it complains with:

ERROR: unable to publish: unable to figure out list of architectures, please supply explicit list

No matter what arguments or which order I pass to aptly:

# aptly publish snapshot -distribution="quantal" -architecture="" 20140318-stable
flag provided but not defined: -architecture
# aptly publish snapshot -distribution="quantal" -architecture="amd64" 20140318-stable
flag provided but not defined: -architecture
# aptly publish snapshot -distribution="quantal" -architecture="[amd64]" 20140318-stable
flag provided but not defined: -architecture
# aptly publish snapshot -architecture=amd64 -distribution="quantal"  20140318-stable
flag provided but not defined: -architecture

Finally I came up with the solution of setting the architectures in the config file (attached below). Is this a bug, or is a documentation defect or may be I'm wrong with something?

{
  "rootDir": "/var/www/repository",
  "downloadConcurrency": 4,
  "architectures": ["amd64","i386"],
  "dependencyFollowSuggests": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowAllVariants": false,
  "dependencyFollowSource": false,
  "gpgDisableSign": true,
  "gpgDisableVerify": true,
  "downloadSourcePackages": false,
  "ppaDistributorID": "ubuntu",
  "ppaCodename": ""
}

Adding packages built from gcc-defaults (1.77) source creates non standard folder in pool

When you add e.g. the package g++_4.3.2-2_i386.deb to a repo and subsequently publishing a snapshot, the package is put in a folder named "gcc-defaults (1.77)", while in the official mirrors it is put in a folder named "gcc-defaults".

The control file for that package says:

Source: gcc-defaults (1.77)

Details

Debian official mirror:

http://ftp.ca.debian.org/debian/pool/main/g/gcc-defaults/

Aptly:

ne@seekyou:~/src/aptly$ aptly repo create testrepo
Config file not found, creating default config at /home/me/.aptly.conf


Local repo [testrepo] successfully added.
You can run 'aptly repo add testrepo ...' to add packages to repository.


me@seekyou:/tmp$ aptly repo add testrepo g++_4.3.2-2_i386.deb 
Loading packages...
[+] g++_4:4.3.2-2_i386 added


me@seekyou:/tmp$ aptly snapshot create testsnapshot from repo testrepo

Snapshot testsnapshot successfully created.
You can run 'aptly publish snapshot testsnapshot' to publish snapshot as Debian repository.


me@seekyou:/tmp$ aptly publish snapshot -distribution="test" testsnapshot
Loading packages...
Generating metadata files and linking package files...
Signing file '/home/me/.aptly/public/dists/test/Release' with gpg, please enter your passphrase when prompted:
Clearsigning file '/home/me/.aptly/public/dists/test/Release' with gpg, please enter your passphrase when prompted:

Snapshot testsnapshot has been successfully published.
Please setup your webserver to serve directory '/home/me/.aptly/public' with autoindexing.
Now you can add following line to apt sources:
  deb http://your-server/ test main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.


me@seekyou:/tmp$ tree ~/.aptly/public/
/home/me/.aptly/public/
├── dists
│   └── test
│       ├── InRelease
│       ├── main
│       │   └── binary-i386
│       │       ├── Packages
│       │       ├── Packages.bz2
│       │       └── Packages.gz
│       ├── Release
│       └── Release.gpg
└── pool
    └── main
        └── g
            └── gcc-defaults (1.77)
                └── g++_4.3.2-2_i386.deb

8 directories, 7 files


me@seekyou:/tmp$ cat ~/.aptly/public/dists/test/main/binary-i386/Packages
Package: g++
Version: 4:4.3.2-2
Installed-Size: 40
Priority: optional
Section: devel
Maintainer: Debian GCC Maintainers <[email protected]>
Architecture: i386
Description: The GNU C++ compiler
 This is the GNU C++ compiler, a fairly portable optimizing compiler for C++.
 .
 This is a dependency package providing the default GNU C++ compiler.
MD5sum: b9286990012bcd60b2c697ae2a5d2241
SHA1: f275b15a270c933d4185cc9f891d8951ef22a43d
SHA256: c991188bbb7a9b77ced97472a5ef03340419c001bd1537cc40b05cc326ca9959
Suggests: g++-multilib
Provides: c++-compiler
Source: gcc-defaults (1.77)
Filename: pool/main/g/gcc-defaults (1.77)/g++_4.3.2-2_i386.deb
Size: 1368
Depends: cpp (>= 4:4.3.2-2), gcc (>= 4:4.3.2-2), g++-4.3 (>= 4.3.2-1), gcc-4.3 (>= 4.3.2-1)

Reprepro:

root@seekyou:/tmp# reprepro -b /var/www/reprepro/ includedeb test g++_4.3.2-2_i386.deb 
Exporting indices...


root@seekyou:/tmp# tree /var/www/reprepro/pool/main/g/
/var/www/reprepro/pool/main/g/
└── gcc-defaults
    └── g++_4.3.2-2_i386.deb

1 directory, 1 file


root@seekyou:/tmp# cat /var/www/reprepro/dists/test/main/binary-i386/Packages
Package: g++
Source: gcc-defaults (1.77)
Version: 4:4.3.2-2
Architecture: i386
Maintainer: Debian GCC Maintainers <[email protected]>
Installed-Size: 40
Depends: cpp (>= 4:4.3.2-2), gcc (>= 4:4.3.2-2), g++-4.3 (>= 4.3.2-1), gcc-4.3 (>= 4.3.2-1)
Suggests: g++-multilib
Provides: c++-compiler
Priority: optional
Section: devel
Filename: pool/main/g/gcc-defaults/g++_4.3.2-2_i386.deb
Size: 1368
SHA256: c991188bbb7a9b77ced97472a5ef03340419c001bd1537cc40b05cc326ca9959
SHA1: f275b15a270c933d4185cc9f891d8951ef22a43d
MD5sum: b9286990012bcd60b2c697ae2a5d2241
Description: The GNU C++ compiler
 This is the GNU C++ compiler, a fairly portable optimizing compiler for C++.
 .
 This is a dependency package providing the default GNU C++ compiler.

forking aptly

Hi Andrey,

I tried to fork your github repository to experiment, but I encountered some difficulties especially about imports...

for example:
import (
"fmt"
"github.com/smira/aptly/aptly"
"github.com/smira/aptly/deb"
"github.com/smira/commander"
"github.com/smira/flag"
"os"
"time"
)

the "github.com/smira/aptly/aptly"
"github.com/smira/aptly/deb"

should be independent from the repository, something like:
"github.com/$USER/aptly/aptly"
"github.com/$USER/aptly/deb"

or do you have a better idea, otherwise it will be a hurdle at the time of pull request...

Eliminate homegrown escaping for gographviz

http://code.google.com/p/gographviz/source/browse/escape.go?spec=svn255110db6f2fb3452cb196d112048f3a15526c98&r=255110db6f2fb3452cb196d112048f3a15526c98

uniform aptly publish list and aptly snapshot list output...

the list command differently outputs repo name:

aptly publish list                                                                                                                                                                                                                                                 
Published repositories:
  * ./precise-security (main) [amd64, i386, source] publishes [precise-security-daily]: Snapshot from mirror [precise-security]: http://de.archive.ubuntu.com/ubuntu/ precise-security [src]
  * ./precise-updates (main) [amd64, i386, source] publishes [precise-updates-daily]: Snapshot from mirror [precise-updates]: http://de.archive.ubuntu.com/ubuntu/ precise-updates [src]

aptly snapshot list                                                                                                                                                                                                                                                
List of snapshots:
 * [precise-backports-daily]: Snapshot from mirror [precise-backports]: http://de.archive.ubuntu.com/ubuntu/ precise-backports [src]
 * [precise-daily]: Snapshot from mirror [precise]: http://de.archive.ubuntu.com/ubuntu/ precise [src]
 * [precise-security-daily]: Snapshot from mirror [precise-security]: http://de.archive.ubuntu.com/ubuntu/ precise-security [src]
 * [precise-updates-daily]: Snapshot from mirror [precise-updates]: http://de.archive.ubuntu.com/ubuntu/ precise-updates [src]

would be nice for scripting to uniform output data
to [precise-backports-daily] for example...

using the -raw argument as implemented in master would do the work, but publish list has no -raw

./aptly publish list -raw
flag provided but not defined: -raw

Best regards

aptly publishing using Debian signature (debian-archive-keyring)

Hi Andrey,

When mirroring an existing archive repository, It would be nice to preserve the Debian signature for the packages matching the debian-archive-keyring public key.

currently as explained in the aptly.info

Command publish
Publishing snapshot as Debian repository which could be served by HTTP/FTP/rsync server. Repository is signed by user's key with GnuPG. Key should be created beforehand (see section GPG Keys). Published repository could be consumed directly by apt.

the user has to generate a GPG key for signing the published packages with a new key.

is the option gpgDisableSign in .aptly.conf configuration or --skip-signing intended to use the original Debian signature of the mirror?

Debian Security Updates

Hello,

I've been trying to mirror Debian Security repository and I have problems.

I create the mirror with this command:
$ aptly -architectures="i386" mirror create -with-sources wheezy-security http://security.debian.org/ wheezy/updates updates/main

I've proved other alternatives, but I think this is the correct one, and it runs without problems, but when I try to update the mirror:
$ aptly mirror update wheezy-security
Downloading http://security.debian.org/dists/wheezy/updates/InRelease...
gpgv: Signature made ...
gpgv: Good signature from ...
Downloading & parsing package files...
Downloading http://security.debian.org/dists/wheezy/updates/updates/main/binary-i386/Packages.bz2...

Obviusly the last URL is not correct, the correct one should be http://security.debian.org/dists/wheezy/updates/main/binary-i386/Packages.bz2, I think.

Thanks and regards.

Allow command line parameters to specified anywhere in the command

It is confusing that command line parameters have to specifed in an exact location in the command. In

aptly publish snapshot name

if you want to specify the distribution name, you have to do

aptly publish snapshot -distribution="wheezy" name

You can't do

aptly publish snapshot name -distribution="wheezy"

or

aptly -distribution="wheezy" publish snapshot name

I recently went through this:

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~/.aptly$ aptly publish snapshot pgdbsc
ERROR: unable to guess distribution name, please specify explicitly

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~/.aptly$ aptly publish snapshot pgdbsc -distribution="wheezy"
ERROR: unable to guess distribution name, please specify explicitly

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~/.aptly$ aptly -distribution="wheezy" publish snapshot pgdbsc
flag provided but not defined: -distribution

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~/.aptly$ aptly publish -distribution="wheezy" snapshot pgdbsc
aptly publish - manage published repositories

Commands:

drop        remove published repository
list        list of published repositories
snapshot    publish snapshot

Use "publish help " for more information about a command.

Finally, I got it right.

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~/.aptly$ aptly publish snapshot -distribution="wheezy" pgdbsc

Mirrors not updating after 0.5 upgrade. (Behind proxy)

Hi.

First of all, thanks for writing this great tool!

Here is my problem: After updating to 0.5 I cant udpate mirrors anymore. I get connection refused. The repos I'm mirroring is only available through a proxy in our environment. I suspect that aptly 0.5 is not using the $HTTP_PROXY environment variable correctly.

Allow publishing local repositories directly

E.g. adding aptly publish repo -distribution=foo <some-repo-name>

Taking snapshots of development/testing repositories is tedious and accumulates a lot of junk.

(One could even default the distribution name to the repository name.)

Compact LevelDB in `aptly db cleanup`

While cleaning up DB should be compacted to really reclaim free disk space.

Allow publishing a package to an existing repo via REST

We would love to be able to build a debian package in Jenkins, and publish the package to our repo server via CURL to a REST endpoint. Any thoughts on adding support for REST-based management of the Aptly service?

debian/ directory should be reserved for packaging

I started looking at packaging aptly for Debian and noticed that the debian/ directory is used for something other than packaging files. This complicates packaging aptly significantly, and is a pretty ironic choice for a project meant for handling debian packages.

We should move what's in debian/ now to something else, maybe src/.

Support Caching Apt Proxying

Aptly looks like an excellent project, there is only one thing stopping us @infoxchange from using it - It needs support for caching external apt repos.

We currently use Approx + Reprepro - which works, but Approx is very buggy and often has cache corruption problems.

aptly db cleanup doesn't remove empty directories

Running aptly db cleanup after removing a large mirror leaves a lot of empty directories around in $rootDir/pool (which the FS spends time digging around in).

I fixed it locally by running find ~/.aptly/pool -type d -name \?\? | sort -r | xargs rmdir, but it would be nice to have aptly figure out by itself.

db corruption, after aptly serve PC crash

I was running aptly serve and froze my PC with another executable, had to do a hard reboot of my machine, but as a result, I get the following error with any aptly call:

aptly mirror update debian-main
ERROR: can't open database: leveldb/storage: corrupted or incomplete CURRENT file

Is there a simple way to fix this?

aptly db cleanup should show disk space freed in units that scale with the amount freed

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~$ aptly db cleanup
Loading mirrors, local repos and snapshots...
Loading list of all packages...
Deleting unreferenced packages (0)...
Building list of files referenced by packages...
Building list of files in package pool...
Deleting unreferenced files (1)...
Disk space freed: 0.00 GiB...

Instead of saying 0.00 GiB when the amount freed is less than 10 MB, it would be nice to see 567.89 KiB.

Error installing aptly on debian 7 amd64

root@vm84451:~# go get github.com/smira/aptly
# github.com/syndtr/goleveldb/leveldb
/usr/lib/go/src/pkg/github.com/syndtr/goleveldb/leveldb/session_util.go:28: method d.file.Num is not an expression, must be called
# github.com/ugorji/go/codec
/usr/lib/go/src/pkg/github.com/ugorji/go/codec/encode.go:107: undefined: io.ByteWriter

devopsru ;)

"aptly help repo create" creates a config file

lhcgreg@debian-70rc1-x64-vbox4210-nocm:~$ aptly help repo create
Config file not found, creating default config at /home/lhcgreg/.aptly.conf

Usage: aptly repo create
...

I would not expect a help command to have any side effects.

aptly mirror update with already existing .aptly/pool

Hi Andrey,

aptly version: 0.4

I was experimenting with some aptly bash wrapper scripts:
mirror_create_ubu_clone.sh: cloning ubuntu with one mirror per separate component: precise-main precise-restricted, ..., precise-security-main, ...

starting with an empty .aptly/db directory.

  1 #!/bin/sh
  2 
  3 # depends on aptly (http://aptly.info)
  4 protocol="http://"
  5 url=ch.archive.ubuntu.com/ubuntu
  6 arch="i386,amd64"
  7 sections="main restricted universe multiverse"
  8 releases="precise precise-security precise-updates precise-backports"
  9 # create new precise repositories
 10 for release in ${releases}
 11 do
 12   for section in ${sections}
 13   do
 14     echo "aptly -architectures="${arch}"  mirror create ${release}-${session} ${protocol}${url} ${release} ${section}"
 15     aptly -architectures="${arch}"  mirror create ${release}-${section} ${protocol}${url} ${release} ${section}
 16   done
 17 done
 18 # update all precise repositories
 19 aptly mirror list | sed -e 's#.*\[\(.*\)\]:.*#\1#g' | grep -E 'precise.*' | xargs -n 1 aptly mirror update

I was hoping that aptly could use the already downloaded packages present in the .aptly/pool/

after some download time, the last line updating all the existing precise repositories does end.

after what I call aptly db cleanup:

aptly db cleanup 
Loading mirrors, local repos and snapshots...
Loading list of all packages...
Deleting unreferenced packages (11528)...
Building list of files referenced by packages...
Building list of files in package pool...
Deleting unreferenced files (11779)...
Disk space freed: 33.06 GiB...

using the same line 19 from the script: started to download 30 GiB again...

after the update for all mirror finished I restarted the command once again, and it still re downloaded some package since the size seems to be different!

Publishing multiple local repositories as different components

It doesn't seem to be possible to publish multiple local repositories as different components within the same distribution, ex.

aptly publish snapshot -distribution dev -component stable dev-repo-20140301
aptly publish snapshot -distribution dev -component testing dev-repo-20140314
ERROR: prefix/distribution already used by another published repo: ...

(In general I find the publishing semantics a bit weird. There's also seem to be significant differences between publishing local repositories and mirrors, though I haven't read enough of the code to figure it out yet.)

Ability to "publish" to S3...

Hey guys, we've been waiting for a year to see someone step up to the plate and build a great repository management tool. Thank you for finally doing it.

We host our own packages (internal ones) in Apt repos that are then stored on S3. Using the apt-transport-s3 package, we're able to securely pull down these packages from our systems in the cloud without worrying about scale, performance or mirror reliability.

What do you think about adding in S3 support to the publish command, so that you can manage your Apt repos on your Aptly host, but then explicitly publish them to S3? With a bit of intelligence (pulling down MD5 hashes from S3 and comparing them), individual updates to repos could be quick and easy.

... create repo Foo
... publish repo Foo to S3
... add package Bar to repo Foo
... publish repo Foo to S3
( only package Bar and the associated metadata files change)
... package Bar + metadata files uploaded to S3

Thoughts?

Support Translations

I manage some repositories in Spain and aptly could be really useful, but translations are really needed.

Have you got this feature in the roadmap?

Non-zero exit status for failed 'repo add' action

I'm driving repo automation from a Perl script, and I've run into the issue where the aptly exits normally (0) when the repo add action has failed, making it difficult to implement my Perl wrapper.

For example:

aptly@myserver:~$ aptly repo add mytools-lucid /tmp/mypackage_0.10-5_all.deb 
Loading packages...
[!] Unable to save package mypackage_0.10-5_all: unable to save: mypackage_0.10-5_all, conflict with existing packge
aptly@myserver:~$ echo $?
0
aptly@myserver:~$ aptly version
aptly version: 0.4.1

I was hoping that aptly would exit with non-zero so I could detect failure of the operation.

Encoding issue in man page

I installed aptly using apt on a fresh Ubuntu precise. man aptly gave me this:

Imposible to make snapshot from OBS

Hi, Andrey!

You are a great man and aptly is a great product!

But now i can't make mirror from OBS repos

$  aptly mirror create virool-obs http://repos.express42.com/virool precise
Downloading http://repos.express42.com/dists/precise/Release...
ERROR: unable to fetch mirror: HTTP code 404 while fetching       
http://repos.express42.com/dists/precise/Release

http://repos.express42.com/virool/precise/ - this is a repo

But i can add this to source.list like this:

deb     http://repos.express42.com:/virool/precise/ ./

Document/improve workflows

There's currently quite a few steps involved in uploading/publishing new packages, and it doesn't work out of the box with tools like dput.

I would really appreciate a section in the documentation describing/discussing a aptly setup as a whole.

Longer-term, it would also be nice with some helper-scripts/tools that eg. can take packages from an /incoming queue and put them in specific repositories.

build aptly from source

Hi Andrey,

I was trying out to build the aptly project from the sources, as defined in the Readme.

go version
go version go1.2.1 linux/amd64

unfortunately the following command:
go get -u github.com/mattn/gom

does not provide any gom executable...
"gom install" resulting in:
zsh: command not found: gom

I tried out apt-cache show gom
Package: gom
Priority: optional
Section: universe/sound

but seems to be something else...

Have defaults for publishing

Introduce default distribution/component for local repositories.

Try harder at guessing distribution/component for snapshot when it has deep tree.

Command line options to config file

How i can move all CLI options about mirrors configuration to config file?

Make 'Origin' and 'Label' configurable for local repositories

Hi,

it would be great if Origin and Label would be configurable like in reprepro or freight.
This is quite useful for APT pinning.

Cheers,
Jan

Publishing to an empty repository fails

Publishing an empty repository currently produces an error. This makes it difficult to automate the setup of our aptly repositories (using Puppet) because the repository's skeleton is not created until the first .deb is added. There is a similar issue with source packages: running an apt-get update against a fresh aptly repository fails on a missing Source file until a .dsc file is committed. It would be convenient if aptly created the full directory structure by default.

Also, I just wanted to share that we really appreciate the development and documentation effort you put into Aptly. We've been struggling with the nightmare of debarchiver, mini-dinstall, and reprepro never managed to get a satisfactory solution. We made more progress with aptly in a night than with the other tools in over a month. Thanks!

Lets start support rpm based OS

many users would like to use your tool for RPM repositories

[enhancement] multiple mirror update

aptly mirror list
List of mirrors:

• [debian-main]: http://ftp.ch.debian.org/debian/ wheezy [src]
• [debian-security]: http://security.debian.org/ wheezy/updates [src]

Would it be possible to update multiple repos in one aptly call?
/usr/bin/aptly -config=/home/kellere5/.aptly.conf mirror update debian-main,debian-security
or
/usr/bin/aptly -config=/home/kellere5/.aptly.conf mirror update all

/usr/bin/aptly -config=/home/kellere5/.aptly.conf mirror update [regexp]="debian.*"

Reagards

Incorrect CLI usage should return non-zero exit code

If you supply an invalid command (not flag/option) and the usage message is printed to STDERR, aptly doesn't return a non-zero exit code to indicate a problem:

root@apt-1:~# aptly repo foo
aptly repo - manage local package repositories

Commands:

    add         add packages to local repository
    copy        copy packages between local repositories
    create      create local repository
    drop        delete local repository
    edit        edit properties of local repository
    import      import packages from mirror to local repository
    list        list local repositories
    move        move packages between local repositories
    remove      remove packages from local repository
    show        show details about local repository

Use "repo help <command>" for more information about a command.


Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -dep-follow-all-variants=false: when processing dependencies, follow a & b if depdency is 'a|b'
  -dep-follow-recommends=false: when processing dependencies, follow Recommends
  -dep-follow-source=false: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests=false: when processing dependencies, follow Suggests
root@apt-1:~# echo $?
0

The same is also true if you put your flags/options in the wrong place:

root@apt-1:~# aptly repo create foo -bar
Usage: aptly repo create <name>

aptly repo create - create local repository


Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -comment="": any text that would be used to described local repository
  -component="main": default component when publishing
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -dep-follow-all-variants=false: when processing dependencies, follow a & b if depdency is 'a|b'
  -dep-follow-recommends=false: when processing dependencies, follow Recommends
  -dep-follow-source=false: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests=false: when processing dependencies, follow Suggests
  -distribution="": default distribution when publishing
root@apt-1:~# echo $?
0

Whereas I'd expect an exit code of 2 so that the problem/misuse can be surfaced to other programs that might want to script aptly. Take grep for example:

root@apt-1:~# grep
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
root@apt-1:~# echo $?
2

This doesn't appear to be trivial to fix because the usage functionality is deep within commander which only returns an error if the templating fails. Could be related to using another library in #50.

download progress > 100%

some odd download progress indication...
25.37 GB / 24.67 GB 102.85 % 12.05 MB/s

100%

aptly snapshot pull more than 128 pkgs: "panic: runtime error: makeslice: cap out of range"

I started getting this error (with aptly 0.5.1) after adding a few more repositories to my (rather lengthy) pull list. The problem occurs whenever there are more than 128 packages in the pull list (not dependent on the packages themselves). I'm guessing you hard-coded this limit somewhere, thinking "nobody could ever possibly want to pull more than 128 packages."

My work-around for this bug is to chain multiple pulls of 128 packages or less, using split -l 128 $PKGLISTFILE

$ aptly snapshot pull empty-snapshot updates selected $(<$PKGLISTFILE)
Dependencies would be pulled into snapshot:
    [empty-snapshot]: Created as empty
from snapshot:
    [updates]: Snapshot from mirror [ubuntu-12.04-updates-main]: http://de.archive.ubuntu.com/ubuntu/ precise-updates
and result would be saved as new snapshot selected.
Loading packages (5378)...
Building indexes...
panic: runtime error: makeslice: cap out of range [recovered]
    panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
runtime.panic(0x83cb400, 0x88da31c)
    /home/smira/.gvm/gos/go1.2.1/src/pkg/runtime/panic.c:266 +0xac
main.func·001()
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/main.go:14 +0x7b
runtime.panic(0x83cb400, 0x88da31c)
    /home/smira/.gvm/gos/go1.2.1/src/pkg/runtime/panic.c:248 +0xef
github.com/smira/aptly/cmd.aptlySnapshotPull(0x18c7d630, 0x18c81010, 0x88, 0x88, 0xb75c7d28, ...)
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/cmd/snapshot_pull.go:89 +0xd74
github.com/smira/commander.(*Command).Dispatch(0x18c7d630, 0x18c81010, 0x88, 0x88, 0x1, ...)
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:299 +0x37a
github.com/smira/commander.(*Command).Dispatch(0x18c7d870, 0x18c81008, 0x89, 0x89, 0x1, ...)
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:277 +0x12c
github.com/smira/commander.(*Command).Dispatch(0x18c7dd80, 0x18c81000, 0x8a, 0x8a, 0x18c7ce40, ...)
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:277 +0x12c
main.main()
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/main.go:34 +0x112

goroutine 9 [sleep]:
time.Sleep(0xbebc200, 0x0)
    /home/smira/.gvm/gos/go1.2.1/src/pkg/runtime/time.goc:31 +0x3c
github.com/cheggaaa/pb.(*ProgressBar).writer(0x18c9a690)
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:263 +0xa4
created by github.com/cheggaaa/pb.(*ProgressBar).Start
    /home/smira/.gvm/pkgsets/go1.2.1/global/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:85 +0x7a

Cannot query aptly while mirror updates

Updating a mirror can take quite some time, initial updates can take hours.

During this time several aptly query commands cannot be used.

This all seems to boil down to the database being locked.

One idea might be to allow (host local) communication with the long running process instead of locking out the user.

Can't mirror jenkins repository

The repository is flat, but it has a component named binary:

$ aptly -architectures="amd64" mirror create jenkins http://pkg.jenkins-ci.org/debian-stable/ ./ binary
ERROR: unable to create mirror: components aren't supported for flat repos

$ aptly -architectures="amd64" mirror create jenkins http://pkg.jenkins-ci.org/debian-stable/binary ./
$ aptly mirror update jenkins
...
ERROR: unable to update: download errors:
  HTTP code 404 while fetching http://pkg.jenkins-ci.org/debian-stable/binary/binary/jenkins_1.554.1_all.deb

Debian Security Updates

Hello,

I've been trying to mirror Debian Security repository and I have problems.

I create the mirror with this command:
$ aptly -architectures="i386" mirror create -with-sources wheezy-security http://security.debian.org/ wheezy/updates updates/main

I've proved other alternatives, but I think this is the correct one, and it runs without problems, but when I try to update the mirror:
$ aptly mirror update wheezy-security
Downloading http://security.debian.org/dists/wheezy/updates/InRelease...
gpgv: Signature made ...
gpgv: Good signature from ...
Downloading & parsing package files...
Downloading http://security.debian.org/dists/wheezy/updates/updates/main/binary-i386/Packages.bz2...

Obviusly the last URL is not correct, the correct one should be http://security.debian.org/dists/wheezy/updates/main/binary-i386/Packages.bz2, I think.

Thanks and regards.

Repository publish automation. passing --passphrase-fd to gpg

Hello,

First of all many thanks for a great project!

We are trying to use aptly in our CI process and have problem with automated repository publishing. aptly publish repo requires entering gpg secret phrase in an interactive mode.

Currently we've worked around it by adding -skip-signing and sign release by calling gpg after publishing. But it'll be better to have ability to pass additional gpg options into aptly publish or have -gpg-passphrase-fd key in aptly publish repo with the same meaning as in gpg.

aptly fails to build on Ubuntu Precise

> $GOROOT/bin/gom install
downloading code.google.com/p/go-uuid/uuid
downloading code.google.com/p/go.crypto/ssh/terminal
downloading code.google.com/p/gographviz
downloading code.google.com/p/snappy-go/snappy
downloading github.com/cheggaaa/pb
downloading github.com/gonuts/commander
downloading github.com/gonuts/flag
downloading github.com/mkrautz/goar
downloading github.com/syndtr/goleveldb/leveldb
downloading github.com/ugorji/go/codec
downloading github.com/wsxiaoys/terminal/color
downloading github.com/golang/lint/golint
downloading github.com/mattn/goveralls
downloading github.com/axw/gocov/gocov
downloading code.google.com/p/go.tools/cmd/cover
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
# github.com/syndtr/goleveldb/leveldb/journal
journal/journal.go:163: function ends without a return statement
# github.com/syndtr/goleveldb/leveldb/memdb
memdb/memdb.go:171: function ends without a return statement
# github.com/syndtr/goleveldb/leveldb/storage
storage/file_storage.go:430: function ends without a return statement
gom:  exit status 2

> go version
go version go1

-> dpkg -l golang
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                     Version                  Description
+++-========================-========================-================================================================
ii  golang                   2:1-5                    Go programming language compiler - metapackage

add ftp support for pulling mirrors

My hosting provider (Strato AG) has a local cache of Ubuntu repositories, but unfortunately only available via ftp, not http. There may be other providers that have similar setup. If aptly could support ftp: as a protocol for pulling mirrors, it would speed up my updates significantly.

This isn't urgent, but perhaps could make it onto your roadmap for aptly.

Thanks for providing such a useful tool.

debootstap fails with repository published by aptly

Split from #44.

@dupuy reports:

I was trying to use debootstrap with an aptly published repository and getting completely obscure errors during the package validation stage:

/usr/sbin/debootstrap: 626: /usr/sbin/debootstrap: arithmetic expression: expecting EOF: "2255928 + 97b1f7e0e3188ba4acf665c3c31d0c2cca5af2665b6b634357ba90bbc0487341 69658"

(debootstrap tries to parse the package information in its shell script, and the spurious spaces threw it off).

Try out cobra instead of commander

https://github.com/spf13/cobra