esell/deb-simple

A lightweight, bare-bones apt repository server

MAINTAINER WANTED

As I am no longer using deb-simple in my day-to-day life, it has been mostly ignored as you can likely see from the issues/PRs. If you are interested in taking over and becoming the maintainer of deb-simple, please open an issue and we can start talking about transitioning it.

deb-simple (get it? dead simple.. deb simple...)

A lightweight, bare-bones apt repository server.

Purpose

This project came from a need I had to be able to serve up already created deb packages without a lot of fuss. Most of the existing solutions I found were either geared at mirroring existing "official" repos or for providing your packages to the public. My need was just something that I could use internally to install already built deb packages via apt-get. I didn't care about change files, signed packages, etc. Since this was to be used in a CI pipeline it had to support remote uploads and be able to update the package list after each upload.

What it does:

• Supports multiple versions of packages
• Supports multi-arch repos (i386, amd64, custom, etc)
• Supports uploading via HTTP/HTTPS POST requests
• Supports removing packages via HTTP/HTTPS DELETE requests
• Does NOT require a changes file
• Supports uploads from various locations without corrupting the repo
• Supports API keys to protect who can upload/delete packages
• Supports signing package release files

What it doesn't do:

• Create actual packages
• Mirror existing repos

General Usage:

This project is now using the native Go vendoring feature so you will need to build with Go >1.7 or if using 1.5/1.6 you will need to make sure GO15VENDOREXPERIMENT is set to 1.

If you do not want to build from source you can just download a pre-built binary from the Releases section.

Fill out the conf.json file with the values you want, it should be pretty self-explanatory, then fire it up!

Once it is running POST a file to the /upload endpoint:

curl -XPOST 'http://localhost:9090/upload?arch=amd64&distro=stable&section=main' -F "[email protected]"

Or delete an existing file:

curl -XDELETE 'http://localhost:9090/delete' -d '{"filename":"myapp.deb","distroName":"stable","arch":"amd64", "section":"main"}'

To use your new repo you will have to add a line like this to your sources.list file:

deb http://my-hostname:listenPort/ stable main

my-hostname should be the actual hostname/IP where you are running deb-simple and listenPort will be whatever you set in the config. By default deb-simple puts everything into the stable distro and main section but these can be changed in the config. If you have enabled SSL you will want to swap http for https.

Package Signing

deb-simple can sign the package release file for you, which will stop apt-get from complaining about insecure sources when you update. To do this you need to enable it in the config file by setting enableSigning to true, and privateKey to the path to your GPG signing key.

If you don't have an existing key deb-simple can help generate one for you. Run:

./deb-simple -k -kn "My Name" -ke "[email protected]"

This will produce two files in the current directory: public.key and private.key. I suggest putting public.key in the repository root somewhere so it can be downloaded by clients that need it, and putting private.key somewhere relatively secure on the file system.

To add your new key on a client run the following command:

wget -qO - http://my-hostname:listenPort/public.key | sudo apt-key add -

This uses Go's native openpgp library, so key support is cross platform, and doesn't require or interact with any existing keyring on the system.

Using API keys:

deb-simple supports the idea of an API key to limit who can upload and delete packages. To use the API keys feature you first need to enable it in the config file by setting enableAPIKeys to true. Once that is done you'll need to generate at least one API key. To do that just run deb-simpled -g and an API key will be printed to stdout.

Now that you have a key you'll need to include it in your POST and DELETE requests by simply adding on the key URL parameter. An example for an upload might look like:

curl -XPOST 'http://localhost:9090/upload?arch=amd64&distro=stable&section=main&key=MY_BIG_API_KEY' -F "[email protected]"

A delete would look like:

curl -XDELETE 'http://localhost:9090/delete?key=MY_BIG_API_KEY' -d '{"filename":"myapp.deb","distroName":"stable","arch":"amd64", "section":"main"}'

If you want an automatable service which builds you packages, either manualy or via CI/CD, checkout debpkg, which makes it very easy to create complex packages with almost no work.

If you want to continuous deliver created packages to deb-simple server, it is not recommended to place the key somewhere others could find it. You can use deb-simple-cd-helper, which allows you to place a plaintext file with the api key somewhere on your build server without the need to expose it.

Directory Watching

By default deb-simple will watch the directories it creates for any new files and rebuild the repository accordingly. This means that you don't have to use the HTTP interface to upload new packages if you have a different build system - any method of getting them onto the server will work.

This function means that there is a delay between a package being uploaded / created and it being availble for installation, as the repository rebuild happens asynchronously. This can result in errors like Hash Sum Mismatch from apt install processes if you happen to update in the middle of a rebuild.

You can disable the watching behaviour by setting enableDirectoryWatching=false in the conf.json file. In this case the repository will be rebuilt as part of the HTTP file upload process, so once your CI build / curl upload has completed the package will be ready for installation.

Do you use this?

If you use deb-simple somewhere I'd love to hear about it! Make a PR to add your company/group/cult :)

• ASE ASE uses deb-simple to serve their FLUID application resources to devices in the wild :)

Contributors

• icholy
• dig412
• tystuyfzand
• kshvakov
• pkramme
• alexanderturner

License:

MIT so go crazy

deb-simple's Issues

Mention building of debian packages with `debpkg`

Hi,

I would like to mention I created a golang package to debianize files and directories. It is not completely finished but it is already functional.

https://github.com/xor-gate/debpkg

Let me know what you think and maybe you can point me to the right direction or contribute if you would like to (when there is a use-case).

Usage as CLI tool

This is a great project, thanks for making it!

Have you considered adding support for using this as a CLI tool? (Or would you accept a PR adding this mode?)

What I mean by that is I would love to be able to run ./deb-simple batch <dir full of deb files> and have it generate all the Packages, Release, GPG-signed files, etc. I would then host the generated files on some static hosting, e.g. S3, GitHub Pages, etc.

In practice, I would run it as part of the CI pipeline for one of my projects, after goreleaser generates the deb files. goreleaser doesn't plan on doing this work itself in the near future: goreleaser/goreleaser#912

no Package: header / Problem with MergeList

Hi,

I currently looking into deb-simple as it looks promising. Unfortunately I encounter a problem that I can't circumvent.

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/localhost:9090_dists_stable_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

my conf.json looks like the default one

{
    "listenPort" : "9090",
    "rootRepoPath" : "/opt/mirror",
    "supportedArch" : ["amd64"],
    "distroNames" : ["stable"],
    "sections" : ["main"],
    "enableSSL" : true,
    "SSLcert" : "server.crt",
    "SSLkey" : "server.key",
    "enableAPIKeys" : false,
    "enableSigning" : true,
    "privateKey" : "./private.key"
}

I generated the private/public key with -k -ke -kn option. The SSLcert is just a snakeoil from my ubuntu.

After initial start I upload some test files from my current deb mirror via for file in $(find . -name *_amd64.deb); do curl -k -X POST 'https://localhost:9090/upload?arch=amd64&distro=stable&section=main' -F "file=@$file"; done. I don't get an error message on upload from curl, nor from the server output (started with -v flag).

My /etc/apt/sources.list contains deb [arch=amd64] https://localhost:9090/ stable main.

How to prevent the error from happening? I already tried sudo rm /var/lib/apt/lists/* -vf without success.

Changes from ubuntu 16.04 to 18.04

It seems that Bionic Beaver (18.04) requires more fields than Xenial (16.04) in the Packages.gz file...
This worked on Xenial:

Filename: dists/xenial/main/binary-amd64/<name>_<version>_amd64.deb
Size: <size>
MD5sum: <md5>
SHA1: <sha>
SHA256: <sha>

But does not work on Bionic; I added these lines to the extracted file at /var/lib/apt/lists/:

Package: <name>
Architecture: amd64
Version: <version>
Priority: standard
Description: <description>

And it worked fine.

This may be an issue with my debian packaging tool, not necessarily an issue with deb-simple, but just thought you might want to know.

deb-simple API client

When integrating with other golang software. deb-simple could benefit from an API client package. E.g:

package main

import (
    "github.com/esell/deb-simple/client"
)

func main() {
    client.New(&client.Config {
     APIKey: "1337",
    })
    client.PublishDebFile("myfile.deb")
}

Handle trailing whitespace in control files

In func inspectPackageControl , we return the args via return controlBuf.String(), nil. However, I've encountered control files that have a trailing newline. The return string from this func is used to create the 'Packages.gz' file, and if the return string has newlines, these make their way into the Packages.gz file but apt clients (esp. Ubuntu 12.04) can get unhappy about stray newlines.

The fix is to trim trailing blank chars from the controlBuf.String()

Access control?

Is there a way to use this in a global production environment by blocking add/delete requests?

Look at memory usage

Right now uploads are stored in memory before being written out to disk. If you had a bunch of uploads going for large packages this could potentially be an issue.

Might be worthwhile to research other possible methods/options for uploading files.

Why is there no deb repository for this project?

As this project is about a debian package server, is there a reason why there is none for this project? Could you create one?

Update documentation

Add some documentation around how you would add your repo into the distro's sources.

Make fsnotify configurable

We've run into an issue using deb-simple where new packages can't be installed until a couple of minutes after uploading them. We have quite a few version of ~50Mb packages, so it takes a little while to hash them all to produce the Packages file.

One potential fix for this would be to move back to the pre-fsnotify behaviour and rebuild the Packages file as part of the package upload handler. This would mean our CI build wouldn't count as complete until the package was ready to install, which makes sense.

I can understand that being able to drop files into the packages directory is useful behaviour for some users, so I don't want to remove it.

@esell what do you think about making the behaviour here configurable? A setting to control if either the directory should be watched, or the Packages file should be rebuilt as a blocking part of the upload.

MAINTAINER WANTED

As I am no longer using deb-simple in my day-to-day life, it has been mostly ignored as you can likely see from the issues/PRs. If you are interested in taking over and becoming the maintainer of deb-simple, please let me know

Ability to use a directory containing deb pkgs

Most times we already have a directory full of packages that we'd like to have served via deb-simple. Is there an easy way to add a config flag that would allow this dir to be served instead of iterating over every file in the dir and POSTing it to the server (which takes a long time when the files are too many or too large).

Happy to take on the work if you think this is functionality you agree with (and have thoughts on how to implement).

Create additional createPackagesGz() test

Currently only testing for the "stable" distro so a bug was missed (#10) where createPackagesGz() was only using the default value. Create another test for non-default distros.

Where are the binaries?

If you do not want to build from source you can just download a pre-built binary from the Releases section.
Uhm, there is no release. Could you make one?

Illegal option -s

I've tried to build debian package in docker container golang:1.12-buster and hit an error

root@3da6c33c12a8:/go/src/github.com/esell/deb-simple# make build-deb
which -s dpkg-deb || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
Illegal option -s
Usage: /usr/bin/which [-a] args
dpkg-deb does not exist, exiting...
make: *** [Makefile:26: build-deb] Error 1

here is my patch to fix it

index 3976ed9..548c147 100644
--- a/Makefile
+++ b/Makefile
@@ -23,8 +23,7 @@ build-linux:
 build-osx:
 	GOOS=darwin go build -o release/$(BINARY)-$(VERSION)-osx
 build-deb:
-	which -s dpkg-deb || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
-	mkdir release/$(BINARY)-$(VERSION)
+	dpkg-deb --version >/dev/null || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
 	mkdir -p release/$(BINARY)-$(VERSION)/usr/local/bin
 	mkdir -p release/$(BINARY)-$(VERSION)/etc/deb-simple
 	cp -r DEBIAN release/$(BINARY)-$(VERSION)/

Add support for Let's Encrypt

Would be nice to support auto creation of Let's Encrypt certs

Add ability to set distro and section

Right now everything is put into the "stable" distro and the "main" section. Would be nice to be able to change these defaults.