cybozu-go/aptutil

Go utilities for Debian APT repositories

go-apt-cacher is a caching reverse proxy built specially for Debian (APT) repositories.
This repository also contains a mirroring utility go-apt-mirror.

Blog: Introducing go-apt-cacher and go-apt-mirror

Status

This project is currently in maintenance mode and is not expected to receive feature updates. Bugfixes and such will still be performed if needed.

Features

go-apt-cacher

• Checksum awareness
  go-apt-cacher recognizes APT indices and checks downloaded files automatically.
• Reverse proxy for http and https repositories
• LRU-based cache eviction
• Smart caching strategy specialized for APT

go-apt-mirror

• Atomic update of mirrors
  Clients will never see incomplete/inconsistent mirrors.
• Checksum validation of mirrored files
• Ultra fast update compared to rsync
• Parallel download
• Partial mirror

Install

Pre-built binaries are available on releases.

Unpack one and follow usage instructions.

Usage

• go-apt-cacher
• go-apt-mirror

Deploy

Both go-apt-cacher and go-apt-mirror can be deployed via the jacksgt/aptutil Docker Image. For more information head over to the source repository.

Build

Use an officially supported version of Go.

Run the command below exactly as shown, including the ellipsis. They are significant - see go help packages.

go get -u github.com/cybozu-go/aptutil/...

License

MIT

Authors & Contributors

• Yamamoto, Hirotaka @ymmt2005
• Yutani, Hiroaki @yutannihilation
• @xipmix
• @jacksgt
• @arnarg

aptutil's Issues

Official docker image

Over at https://github.com/jacksgt/docker-aptutil I have created a Dockerfile for aptutil.

It includes both go-apt-cacher and go-apt-mirror.

---

Usage is rather simple:

go-apt-mirror

Mount the configuration file for go-apt-cacher to /etc/go-apt-cacher.toml (read-only for added security).

Optionally, mount /var/spool/go-apt-cacher for persistence (required for --read-only).

$ docker container run \
    --read-only \
    -v /etc/go-apt-cacher.toml:/etc/go-apt-cacher.toml:ro \
    -v /var/spool/go-apt-cacher:/var/spool/go-apt-cacher \
    jacksgt/aptutil /go-apt-cacher

go-apt-mirror

Mount the configuration file for go-apt-cacher to /etc/apt/mirror.toml (read-only for added security).

Mount /var/spool/go-apt-mirror to store the mirrored files permanently (required for --read-only).

$ docker container run \
    --read-only \
    -v /etc/go-apt-mirror.toml:/etc/apt/mirror.toml:ro \
    -v /var/spool/go-apt-mirror:/var/spool/go-apt-mirror \
    jacksgt/aptutil /go-apt-mirror

---

Are you interested in merging this into your repository?
If so, it would be great if you created a DockerHub account and set up an automatic build. That way, the image will be automatically built each time someone pushes into the source repo.
By 'linking' against the golang docker image repository you can also ensure the image is rebuilt when there is a new version of the golang image ("Repository Links").
See https://hub.docker.com/r/jacksgt/docker-aptutil/

[cacher] Frequency of Release checks too high?

The default is 15 seconds. My concern is, if this utility comes into widespread use, won't it put a large load on the upstream servers? The whole point of having an apt-cache is to reduce the load on the upstream servers.
Can you say what the rationale for 15s was? What would be the downsides of setting this to, say 3601s?

error when synchronizing mirrors created by aptly

This tools is quite friendly when synchronizing mirrors, I've hit my head for a few days in regards to synchronizing mirrors, Icinga being amongst them.
From the debugging I've done, the common denominator is how aptly does not provide an uncompressed Contents- file, such as Contents-amd64. You can download the file without the file extension as the Release says is available, the Header "Content-Location" is returned with the filename you are really getting (filename.gz), which the net/http library might not handle that well.

Example:
As a consequence when trying to get https://packages.icinga.com/debian/dists/icinga-buster/main/Contents-amd64, you get served https://packages.icinga.com/debian/dists/icinga-buster/main/Contents-amd64.gz while the function checks against the file without compression. If the function checks in case it fails against a .gz entry for the same filename, it would work.

Toml to replicate the error:

dir = "/var/spool/go-apt-mirror"
max_conns = 10

[log]
level = "debug"
format = "plain"

[mirror.icinga]
url = "https://packages.icinga.com/debian"
suites = ["icinga-buster"]
sections = ["main"]
mirror_source = false
architectures = ["amd64"]

How to support security.debian.org?

Problem

It seems that go-apt-cacher can't handle missing path case.
Are there any workaround configuration on this issue?

Expected

There is a workaround configuration in go-apt-cacher.toml or something.

Actual

It works with apt-line:

deb http://security.debian.org/debian-security/ jessie/updates main
deb-src http://security.debian.org/debian-security/ jessie/updates main

and go-apt-cacher.toml.

[mapping]
debian = "http://ftp.jp.debian.org/debian"
debian-security = "http://security.debian.org/debian-security"

But It doesn't work with apt-line and above toml:

deb http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main

% sudo apt update
...(skip)...
Fetched 23.1 MB in 2s (10.4 MB/s)                                                 
W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/source/Sources  404  Not Found

W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

How to configure go-apt-cacher.toml mapping correctly for http://security.debian.org/? (without modifiyng /etc/apt/sources.list?)

does not appear to stream the download to the first client as it's downloading from the upstream

Exporting more functions to allow more code re-use?

For Concourse, I'm currently implementing a new apt-package resource type that can easily monitor the availability of new versions of a certain package. Neither the existing aptly-cli nor the debian-sources resource type fulfils my needs.

This repository contains 90% of the functionality to support me in implementing this, weren't it for the fact that a number of functions in the mirror package are not exported. E.g. I would like to use functions downloadRelease and downloadIndices, possibly some more.

I'm not in favor of duplicating code or maintaining my own fork. Are you open to a PR that contains the changes I need to get my Concourse resource implemented?

Missing files from repositories create neverending sync jobs

Problem report template:

Description

Apparently, there are many broken debian repositories out there, where the "Release file" contains entries, which are missing from the repository.
Some examples like this are some public Zabbix repositories etc.

Reproducible? Yes

Time of the issue

Can happen at any time

System

Company test systems

Software version

go-apt-mirror: v1.4.2

Prerequisites

Configure sync to some repositories, which where there are some 'broken' files

Behavior

Steps to reproduce

Execute go-apt-mirror -f mybrokenrepos.toml

Expected behavior

All available files are synced.

Actual behavior

The sync software, it it encounters an error (file missing from remote repo), will attempt to sync repo again.

Corrective actions taken

Not any

Suggestion for a fix

When calling downloadFiles at:
https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L524
we are setting allowMissing to false. This causes the caller function https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L173 to return error causing the control function get get an error return https://github.com/cybozu-go/aptutil/blob/master/mirror/control.go#L40 stopping the update.

Suggestions: Let's make the setting allowMissing configurable in the .toml file on a per repository basis (but defaulting to false, to preserve the original behavior), and to allow handling the situation gracefully.

transparently proxy repositories

Hi,

One of the advantages of apt-cacher-ng was that sources.list files didn't need updating.
Just the Acquire::http::Proxy statement, or http(s)_proxy enviroment line were required.

When you start adding other repos for software projects, either manually or using PPAs, this looks like it will be messy. How does the proxy know which repo you wish to use?

fatal error: runtime: out of memory

Downloding large package cause go-apt-cacher with "out of memory" error. By execute apt-get download texlive-latex-extra-doc texlive-pstricks-doc

Package: texlive-latex-extra-doc
Download-Size: 384 MB

Package: texlive-pstricks-doc
Download-Size: 248 MB

On 1GB mem VM. Is this intended?

Log attached.

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x6f8ba7, 0x16)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc4427b0000, 0x20000000, 0x0, 0x8621b8)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x848be0, 0x20000000, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x848be0, 0x10000, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x848be0, 0x10000, 0x42f87f)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).alloc_m(0x848be0, 0x10000, 0x100000000, 0xc42014bec8)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:515 +0xe0
runtime.(*mheap).alloc.func1()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:579 +0x4b
runtime.systemstack(0xc42014bed0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:314 +0xab
runtime.(*mheap).alloc(0x848be0, 0x10000, 0x10100000000, 0xc421598990)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/mheap.go:580 +0x73
runtime.largeAlloc(0x1ffffe00, 0xc421598901, 0xc420147990)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:774 +0x93
runtime.mallocgc.func1()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:669 +0x3e
runtime.systemstack(0xc420016000)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:1079

goroutine 18 [running]:
runtime.systemstack_switch()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:252 fp=0xc420147898 sp=0xc420147890
runtime.mallocgc(0x1ffffe00, 0x68d040, 0xc420147901, 0x44fe90)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/malloc.go:670 +0x903 fp=0xc420147938 sp=0xc420147898
runtime.makeslice(0x68d040, 0x1ffffe00, 0x1ffffe00, 0x0, 0x0, 0xc421e88c10)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/slice.go:57 +0x7b fp=0xc420147990 sp=0xc420147938
bytes.makeSlice(0x1ffffe00, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/bytes/buffer.go:198 +0x77 fp=0xc4201479d0 sp=0xc420147990
bytes.(*Buffer).ReadFrom(0xc420147a70, 0x824040, 0xc421e88c00, 0xc420078a00, 0x0, 0x200)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/bytes/buffer.go:170 +0x2a9 fp=0xc420147a40 sp=0xc4201479d0
io/ioutil.readAll(0x824040, 0xc421e88c00, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/io/ioutil/ioutil.go:33 +0x147 fp=0xc420147af0 sp=0xc420147a40
io/ioutil.ReadAll(0x824040, 0xc421e88c00, 0xc421e88c00, 0x824040, 0xc421e88c00, 0xc42203c0d0, 0xc421fd8ba0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/io/ioutil/ioutil.go:42 +0x3e fp=0xc420147b40 sp=0xc420147af0
github.com/cybozu-go/aptutil/cacher.(*Cacher).download(0xc42006c360, 0x8286c0, 0xc421a878c0, 0xc420270245, 0x50, 0xc42276e200, 0xc420a81320)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:310 +0x295 fp=0xc420147f00 sp=0xc420147b40
github.com/cybozu-go/aptutil/cacher.(*Cacher).Download.func1(0x828640, 0xc421e88480, 0x828640, 0xc421e88480)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:256 +0x62 fp=0xc420147f48 sp=0xc420147f00
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc421598420)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86 fp=0xc420147f90 sp=0xc420147f48
runtime.goexit()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420147f98 sp=0xc420147f90
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 1 [chan receive]:
github.com/cybozu-go/cmd.(*Environment).Wait(0xc420040310, 0x0, 0x0)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:95 +0x5a
github.com/cybozu-go/cmd.Wait(0xc4204ad180, 0x0)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/default.go:47 +0x2d
main.main()
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cmd/go-apt-cacher/main.go:52 +0x30f

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/os/signal/signal_unix.go:28 +0x41

goroutine 6 [select, locked to thread]:
runtime.gopark(0x71ada8, 0x0, 0x6f376f, 0x6, 0x18, 0x2)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc420020730, 0x0, 0x18)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc420020730)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086 +0x1

goroutine 7 [chan receive]:
github.com/cybozu-go/cmd.handleSignal.func1(0xc4200123c0, 0xc420040310)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/signal.go:27 +0x5a
created by github.com/cybozu-go/cmd.handleSignal
	/home/travis/gopath/src/github.com/cybozu-go/cmd/signal.go:32 +0x9d

goroutine 9 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947980, 0xc42003e81e, 0x1b, 0xc42001af01)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func1(0x828640, 0xc420947980, 0x828640, 0xc420947980)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:192 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d3600)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 10 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947b80, 0xc420042c4e, 0x25, 0xc42001b700)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947b80, 0x828640, 0xc420947b80)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d36e0)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 11 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947d00, 0xc420012dfe, 0x2e, 0xc42001bf00)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947d00, 0x828640, 0xc420947d00)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d3880)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 12 [select]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintRelease(0xc42006c360, 0x828640, 0xc420947800, 0xc4200434be, 0x1d, 0xc42001c700)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:214 +0x280
github.com/cybozu-go/aptutil/cacher.(*Cacher).maintMeta.func2(0x828640, 0xc420947800, 0x828640, 0xc420947800)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:197 +0x55
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc4208d38a0)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 14 [chan receive]:
github.com/cybozu-go/cmd.(*HTTPServer).wait(0xc4204ad180, 0x828640, 0xc420947f00, 0xc42001bf68, 0xc42001bf78)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:212 +0x6c
github.com/cybozu-go/cmd.(*HTTPServer).(github.com/cybozu-go/cmd.wait)-fm(0x828640, 0xc420947f00, 0x828640, 0xc420947f00)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:208 +0x3e
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc420040310, 0xc420ab2830)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/home/travis/gopath/src/github.com/cybozu-go/cmd/env.go:138 +0xaf

goroutine 15 [chan receive]:
github.com/cybozu-go/cmd.(*HTTPServer).Serve.func1(0xc4204ad180, 0x8281c0, 0xc420022000)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:278 +0x5a
created by github.com/cybozu-go/cmd.(*HTTPServer).Serve
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:280 +0xba

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f1d4a4a4e98, 0x72, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204ad250, 0x72, 0xc422049d88, 0xc42000a1c0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204ad250, 0x825b00, 0xc42000a1c0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4204ad1f0, 0x0, 0x824640, 0xc42204a000)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420022000, 0xc4204ad1b0, 0x1, 0xc422049e50)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).AcceptTCP(0xc420022000, 0xc422049eb0, 0xc422049eb8, 0xc422049ea8)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/tcpsock.go:209 +0x49
github.com/cybozu-go/netutil.TCPKeepAliveListener.Accept(0xc420022000, 0x71a7c0, 0xc42276e000, 0x828700, 0xc421e9abd0)
	/home/travis/gopath/src/github.com/cybozu-go/netutil/tcpkeepalive.go:19 +0x2f
net/http.(*Server).Serve(0xc42054d780, 0x8281c0, 0xc420022000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2273 +0x1ce
github.com/cybozu-go/cmd.(*HTTPServer).Serve.func2(0xc4204ad180, 0x8281c0, 0xc420022000)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:283 +0x42
created by github.com/cybozu-go/cmd.(*HTTPServer).Serve
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:284 +0xf0

goroutine 17 [chan receive]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).Get(0xc42006c360, 0xc420270245, 0x50, 0x50, 0x1, 0xc4201301e0, 0x0)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/cacher.go:434 +0x2aa
github.com/cybozu-go/aptutil/cacher.cacheHandler.ServeHTTP(0xc42006c360, 0x827d00, 0xc42204a0e0, 0xc4201301e0)
	/home/travis/gopath/src/github.com/cybozu-go/aptutil/cacher/handler.go:35 +0x185
github.com/cybozu-go/cmd.(*HTTPServer).ServeHTTP(0xc4204ad180, 0x827f80, 0xc421a6e1a0, 0xc4201300f0)
	/home/travis/gopath/src/github.com/cybozu-go/cmd/http.go:132 +0x232
net/http.serverHandler.ServeHTTP(0xc42054d780, 0x827f80, 0xc421a6e1a0, 0xc4201300f0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc42276e000, 0x828640, 0xc421e881c0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2293 +0x44d

goroutine 24 [select]:
net/http.(*persistConn).readLoop(0xc42000ca00)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1536 +0x9c9
created by net/http.(*Transport).dialConn
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1057 +0x4e9

goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0xc42000ca00)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1641 +0x3bd
created by net/http.(*Transport).dialConn
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/transport.go:1058 +0x50e
2017-10-04T11:36:23.589582Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="93ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.058151219000000004 type="access" url="/debian/dists/jessie/InRelease"
2017-10-04T11:36:23.594537Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="90ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=63066 response_time=0.0030334760000000002 type="access" url="/debian-security/dists/jessie/updates/InRelease"
2017-10-04T11:36:23.632614Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="91ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=2373 response_time=0.00025303500000000003 type="access" url="/debian/dists/jessie/Release.gpg"
2017-10-04T11:36:23.635683Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="96ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=148400 response_time=0.001911944 type="access" url="/debian/dists/jessie/Release"
2017-10-04T11:36:24.311989Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "downloaded and cached" path="debian-security/dists/jessie/updates/main/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.316877Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="97ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=433331 response_time=0.635777987 type="access" url="/debian-security/dists/jessie/updates/main/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.354651Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="94ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=2506 response_time=0.00017602900000000002 type="access" url="/debian-security/dists/jessie/updates/contrib/binary-amd64/Packages.bz2"
2017-10-04T11:36:24.356700Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="95ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=1211 response_time=0.0014163930000000002 type="access" url="/debian-security/dists/jessie/updates/contrib/i18n/Translation-en.bz2"
2017-10-04T11:36:24.363795Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9aef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=231458 response_time=0.0065162020000000005 type="access" url="/debian-security/dists/jessie/updates/main/i18n/Translation-en.bz2"
2017-10-04T11:36:24.372568Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9bef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.000028965 type="access" url="/debian/dists/jessie/main/binary-amd64/Packages.xz"
2017-10-04T11:36:24.373048Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher warning: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=404 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="98ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=19 response_time=0.000019858000000000002 type="access" url="/debian/dists/jessie/contrib/binary-amd64/Packages.xz"
2017-10-04T11:36:24.375672Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="99ef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=38307 response_time=0.002113989 type="access" url="/debian/dists/jessie/contrib/i18n/Translation-en.bz2"
2017-10-04T11:36:24.541312Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9eef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=4582022 response_time=0.164297343 type="access" url="/debian/dists/jessie/main/i18n/Translation-en.bz2"
2017-10-04T11:36:24.873764Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9fef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=9063195 response_time=0.225623379 type="access" url="/debian/dists/jessie/main/binary-amd64/Packages.gz"
2017-10-04T11:36:24.996407Z ubuntu.cs2d52idcfcloud.internal go-apt-cacher info: "cmd: access" http_host="10.6.0.179:3142" http_method="GET" http_status_code=200 http_user_agent="Debian APT-HTTP/1.3 (1.0.9.8.4)" protocol="HTTP/1.1" request_id="9cef0f8b-fcfa-7341-3c12-233a88d3e091" request_size=0 response_size=59182 response_time=0.00016119000000000002 type="access" url="/debian/dists/jessie/contrib/binary-amd64/Packages.gz"

[cacher] Ignore checksum of Release in Release

Some APT repositories have a Release file that contains wrong checksums of Release itself.
As go-apt-cacher fails to handle such repositories, such checksums should be ignored.

Explain build process

Can you explain how to build this more clearly? I've never used go before and the conventions are weird. I tried this

mkdir -p ~/gocode/bin
mkdir ~/gocode/src
cd ~/gocode/src
go get -u github.com/cybozu-go/aptutil
go install -v github.com/cybozu-go/aptutil

and nothing happens. I tried

 go build -v github.com/cybozu-go/aptutil/cacher

and got a face full of errors about missing dependencies.

error: "update failed" ... found no Release/InRelease

About

The following settings will fail.

Version

go-apt-mirror v1.3.2 linux/amd64

Config

# Directory to store mirrored files and other control files.
# The directory must be writable by go-apt-mirror.
dir = "/home/takumin/test"

# Maximum concurrent connections for an upstream server.
# Setting this 0 disables limit on the number of connections.
# Default: 10
max_conns = 10

# log specifies logging configurations.
# Details at https://godoc.org/github.com/cybozu-go/cmd#LogConfig
[log]
level = "info"
format = "plain"

# [mirror.xxx] defines a mirror configuration for a debian repository.
# "xxx" must match this regexp: ^[a-z0-9_-]+$
#
# url:           The repository base URL.
# suites:        List of suites to mirror.  see sources.list(5).
# sections:      List of sections to mirror.  see sources.list(5).
# mirror_source: true to mirror source archives.  Default is false.
# architectures: List of architectures to mirror.  "all" is always mirrored.
[mirror.jenkins]
url = "http://pkg.jenkins.io/debian-stable"
suites = ["binary/"]
[mirror.nvidia-cuda]
url = "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64"
suites = ["/"]
[mirror.nvidia-digits]
url = "http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64"
suites = ["/"]

Result

2017-10-05T13:48:41.478084Z srv go-apt-mirror info: "update starts"
2017-10-05T13:48:41.478292Z srv go-apt-mirror info: "download Release/InRelease" repo="jenkins" suite="binary/"
2017-10-05T13:48:41.478396Z srv go-apt-mirror info: "download Release/InRelease" repo="nvidia-digits" suite="/"
2017-10-05T13:48:41.478509Z srv go-apt-mirror info: "download Release/InRelease" repo="nvidia-cuda" suite="/"
2017-10-05T13:48:41.945865Z srv go-apt-mirror info: "detected by-hash support" repo="nvidia-digits" suite="/"
2017-10-05T13:48:41.946167Z srv go-apt-mirror error: "update failed" error="nvidia-digits: found no Release/InRelease"
2017-10-05T13:48:41.946550Z srv go-apt-mirror error: "nvidia-digits: found no Release/InRelease"

Failed to fetch dep11/by-hash File has unexpected size

With 50appstream in apt.conf.d, the following errors occur (example):

Err:10 http://apt-cacher.foo.org:3141/ubuntu bionic/main DEP-11 48x48 Icons
  File has unexpected size (162304 != 118319). Mirror sync in progress? [IP: 10.1.2.3 3141]
  Hashes of expected file:
   - Filesize:118319 [weak]
   - SHA256:72a445c7d354db04d1ffca6f52f7ab66953a3fa889bb66bf6d3210d3ab100f14
   - SHA1:176f580a423a2fcf3e257f0d9a00713e05545a35 [weak]
   - MD5Sum:bdfd4c6a690b2f9d0790443956b327fd [weak]
  Release file created at: Thu, 26 Apr 2018 23:37:48 +0000

Going directly to the mirror set in the config toml, everything works fine.

Full outputs:
apt-get-update.notok.txt
go-apt-cacher.si-ubuntu.notok.log
apt-get-update.ok.txt

Configs:
go-apt-cacher.toml.txt
50appstream.txt

Tested on Ubuntu 18.04 desktop live iso. (same on 16.04/18.04 desktop if installed with 50appstream)

Build failed on Windows

(I don't think we need to support Windows. This issue is just for the record.)

> go get -u github.com/cybozu-go/aptutil/...
# github.com/cybozu-go/aptutil/mirror
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.Flock
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.LOCK_EX
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:17: undefined: syscall.LOCK_NB
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:23: undefined: syscall.Flock
Desktop\Go\src\github.com\cybozu-go\aptutil\mirror\flock.go:23: undefined: syscall.LOCK_UN

cannot startup with big Packages.gz

caching repos like Kali Linux's (http://kali.download/kali) works just for the first run: after a few apt-get installs pointing to go-apt-cacher, stopping and restarting the service results in an endless loop of this message:

SVC:/var/log/supervisord# tail -f go-apt-cacher*
==> go-apt-cacher1.log <==
2020-05-07T11:27:35.101356Z SVC go-apt-cacher warning: "well: got signal" signal="terminated"
2020-05-07T11:27:36.961230Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:50.858243Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:54.403333Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:27:57.930828Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:01.439917Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:04.932175Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:08.437772Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"
2020-05-07T11:28:11.930364Z SVC go-apt-cacher error: "ExtractFileInfo(kali/dists/kali-rolling/main/binary-arm64/Packages.gz): parser.Read: bufio.Scanner: token too long"

==> go-apt-cacher2.log <==

Provide a workaround to ignore bad indices

Some repositories are badly configured to return wrong contents for uncompressed indices.

For instance, pkg.jenkins.io returned wrong Packages as pointed by @takumin:
#25 (comment)

Zabbix also returned wrong Sources for which go-apt-mirror implements a workaround already:

aptutil/mirror/mirror.go

Lines 243 to 257 in 66c646f

	// WORKAROUND: some (zabbix) repositories returns wrong contents
	// for non-existent files such as Sources (looks like the body of
	// Sources.gz is returned).
	if !m.mc.Source {
	tmpMap := make(map[string][]*apt.FileInfo)
	for p, fil := range indexMap {
	base := path.Base(p)
	base = base[0 : len(base)-len(path.Ext(base))]
	if base == "Sources" {
	continue
	}
	tmpMap[p] = fil
	}
	indexMap = tmpMap
	}

Although these indices are bad, apt or apt-get will work nicely as they look compressed indices
such as Packages.bz2 or Sources.gz first.

To help users of go-apt-mirror, we should provide a general workaround for this problem.

downloaded data is not valid on .xz and .bz2 i18n files

Getting the following errors:

2020-03-21T16:46:05.021453Z apt-cacher go-apt-cacher warning: "downloaded data is not valid" url="http://si.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/i18n/Translation-en.bz2"
2020-03-21T16:46:05.032053Z apt-cacher go-apt-cacher warning: "downloaded data is not valid" url="http://si.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/i18n/Translation-en.xz"

Not building on arm

> docker run -ti --rm resin/armhf-alpine-golang bash
bash-4.4# go get -v -u github.com/cybozu-go/aptutil/...
github.com/cybozu-go/aptutil (download)
github.com/pkg/errors (download)
github.com/cybozu-go/cmd (download)
github.com/cybozu-go/log (download)
github.com/cybozu-go/netutil (download)
Fetching https://golang.org/x/net/netutil?go-get=1
Parsing meta tags from https://golang.org/x/net/netutil?go-get=1 (status code 200)
get "golang.org/x/net/netutil": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/netutil?go-get=1
get "golang.org/x/net/netutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
github.com/spf13/pflag (download)
github.com/spf13/viper (download)
github.com/fsnotify/fsnotify (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/hashicorp/hcl (download)
github.com/magiconair/properties (download)
github.com/mitchellh/mapstructure (download)
github.com/pelletier/go-toml (download)
github.com/spf13/afero (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/BurntSushi/toml (download)
github.com/cybozu-go/aptutil
github.com/hashicorp/hcl/hcl/strconv
github.com/spf13/afero/mem
golang.org/x/text/transform
golang.org/x/sys/unix
github.com/pkg/errors
github.com/cybozu-go/aptutil/apt
github.com/cybozu-go/log
golang.org/x/net/netutil
github.com/cybozu-go/netutil
github.com/spf13/pflag
github.com/hashicorp/hcl/hcl/token
github.com/hashicorp/hcl/hcl/ast
github.com/hashicorp/hcl/hcl/scanner
github.com/hashicorp/hcl/json/token
github.com/hashicorp/hcl/hcl/parser
github.com/hashicorp/hcl/json/scanner
github.com/hashicorp/hcl/json/parser
github.com/hashicorp/hcl/hcl/printer
github.com/fsnotify/fsnotify
github.com/hashicorp/hcl
github.com/mitchellh/mapstructure
github.com/pelletier/go-toml
github.com/magiconair/properties
golang.org/x/text/unicode/norm
github.com/spf13/jwalterweatherman
github.com/spf13/cast
gopkg.in/yaml.v2
github.com/BurntSushi/toml
github.com/spf13/afero
github.com/spf13/viper
github.com/cybozu-go/cmd
github.com/cybozu-go/aptutil/cacher
github.com/cybozu-go/aptutil/mirror
# github.com/cybozu-go/aptutil/mirror
src/github.com/cybozu-go/aptutil/mirror/config.go:9:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/mirror/config.go:168:11: undefined: cmd
src/github.com/cybozu-go/aptutil/mirror/control.go:10:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/mirror/mirror.go:16:2: imported and not used: "github.com/cybozu-go/cmd" as well
# github.com/cybozu-go/aptutil/cacher
src/github.com/cybozu-go/aptutil/cacher/cacher.go:20:2: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/cacher/config.go:3:8: imported and not used: "github.com/cybozu-go/cmd" as well
src/github.com/cybozu-go/aptutil/cacher/config.go:61:6: undefined: cmd
src/github.com/cybozu-go/aptutil/cacher/server.go:6:2: imported and not used: "github.com/cybozu-go/cmd" as well

[cacher] Documentation - reload of config file

I have been looking into writing a systemd unit file for this. One thing that came up was reloading the configuration file after it has been modified. USAGE.md says "go-apt-cacher reads a configuration file at start up."

My question is: after a change to the configuration file, does go-apt-cacher notice the file has been updated and automatically reread it?

Or does one need to restart the service, ie kill the existing process and start it again?

go-apt-cacher panics with base prefix

Hi,

I just found an interesting failure in go-apt-cacher (I'm running the latest master/HEAD version).

Using the default config file (which maps ubuntu to http://archive.ubuntu.com/ubuntu), the server panics when querying http://go-apt-cacher:3142/ubuntu/

$ go-apt-cacher --loglevel debug
2017-12-19T13:19:32.594886Z 1eec099b553c go-apt-cacher debug: "request path" path="ubuntu"
panic: path must has a prefix: ubuntu

goroutine 33 [running]:
github.com/cybozu-go/aptutil/cacher.(*Cacher).download(0xc42017a240, 0x89dcc0, 0xc420420120, 0xc42043c005, 0x6, 0xc42018e200, 0x0)
	/go/src/github.com/cybozu-go/aptutil/cacher/cacher.go:338 +0x142a
github.com/cybozu-go/aptutil/cacher.(*Cacher).Download.func1(0x89dc40, 0xc420400100, 0x89dc40, 0xc420400100)
	/go/src/github.com/cybozu-go/aptutil/cacher/cacher.go:256 +0x62
github.com/cybozu-go/cmd.(*Environment).Go.func1(0xc4200982a0, 0xc42041a060)
	/go/src/github.com/cybozu-go/cmd/env.go:133 +0x86
created by github.com/cybozu-go/cmd.(*Environment).Go
	/go/src/github.com/cybozu-go/cmd/env.go:130 +0xaf

caused by a simple:

curl -i 172.17.0.2:3142/ubuntu/

The panic is caused by the following statement in cacher/cacher.go:L338

t := strings.SplitN(path.Clean(p), "/", 2)
if len(t) != 2 {
    panic("path must has a prefix: " + p)
}

I don't think its adequate behavior to kill the server just because of this. go-apt-cacher should either serve the index.html of http://archive.ubuntu.com/ubuntu/ or just reply with 404.

Avoid consuming too much memory

aptutil/mirror/mirror.go

Line 280 in ec162e2

data, err := ioutil.ReadAll(resp.Body)

go-apt-mirror keeps downloaded data in memory.
This may result in excessive memory usage while downloading debs.

go-apt-cacher has a similar issue.

No cleanup for temporary directories if sync fails for them

Description

Only directories that have been linked to the repo-id are cleaned up.

Reproducible? yes, but hard

Time of the issue

Randomly

System

Own test systems

Software version

go-apt-mirror: v1.4.2

Prerequisites

Not really clear why a sync would fail to an apt repo, but the causes could be unreliable IP connectivity or incorrectly configured upstream apt repos.

Behavior

Steps to reproduce

Start synchronizing an apt repository to which there is bad IP connectivity.

Expected behavior

Should the synchronization fail, a message is displayed (it is) and the incorrectly downloaded apt repository is deleted from the disk.

Actual behavior

The incorrectly and incompletely downloaded apt directories are laying around and slowly taking up disk space, no mechanism takes care of cleaning them.

Corrective actions taken

Delete incomplete downloaded apt directories.

Fix suggestion:

Implement a mechanism into the code that would keep a number of [configurable] temporary downloaded directories, and start wiping the older ones.

token too long

I was using apt-cacher-ng but like you, I found it was unstable and I had intermittent issues. I'm trying to use go-apt-cacher as a satellite cache server. Our main repository is hosted with aptly.

I commented the existing mapping found in your sample config file and added mine:

aptly = "https://aptly.example.com/ubuntu"

The cacher is working fine. The first time a client ask for a package, it downloads it from the main repository and cache it locally. When another client ask for the same file again, the cacher get it from the local cache and doesn't get it from the central repository. All fine!

Problem is when I stop the cacher and try to start again. I get this error:

2020-04-07T20:28:26.549038Z deploiement-ext go-apt-cacher error: "ExtractFileInfo(aptly/dists/focal/main/binary-amd64/Packages.gz): parser.Read: bufio.Scanner: token too long"

The only thing I found to "fix" that is to remove the "meta" directory content. But by doing that, this seem to invalidate my local cache.

[mirror] failed to detect by-hash support for specific configurations

go-apt-mirror can detect by-hash support if all suites provide Acquire-By-Hash: yes in Release files.

But if one suite does not support by-hash while the others do, go-apt-mirror misinterprets that
all suites do not support by-hash. This is not ideal.

[cacher] Listen address in toml file?

It would be nice to be able to set the go-apt-cacher listen address in the .toml file instead of with e.g. -s :9999 on the command line. I can't see this documented anywhere. Is it currently possible, or is new code required?

error= no files in .../Sources.gz

Hey,

I've got an error while mirroring the debian buster repo.

2020-04-16T19:54:42.020106Z goaptmirror go-apt-mirror error: "update failed" error="debian: no Files in dists/buster-updates/main/source/Sources.gz"
2020-04-16T19:54:42.020187Z goaptmirror go-apt-mirror error: "debian: no Files in dists/buster-updates/main/source/Sources.gz"

My mirror.toml:

# Directory to store mirrored files and other control files.
# The directory must be writable by go-apt-mirror.
dir = "/mnt/mirror"

# Maximum concurrent connections for an upstream server.
# Setting this 0 disables limit on the number of connections.
# Default: 10
max_conns = 10

# log specifies logging configurations.
# Details at https://godoc.org/github.com/cybozu-go/well#LogConfig
[log]
level = "info"
format = "plain"
#filename = "go-apt-mirror.log"

# [mirror.xxx] defines a mirror configuration for a debian repository.
# "xxx" must match this regexp: ^[a-z0-9_-]+$
#
# url:           The repository base URL.
# suites:        List of suites to mirror.  see sources.list(5).
# sections:      List of sections to mirror.  see sources.list(5).
# mirror_source: true to mirror source archives.  Default is false.
# architectures: List of architectures to mirror.  "all" is always mirrored.
[mirror.debian]
url = "http://debian.uni-duisburg-essen.de/debian"
suites = ["stretch", "stretch-updates", "buster", "buster-updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]

#[mirror.ubuntu]
#url = "http://archive.ubuntu.com/ubuntu"
#suites = ["bionic", "bionic-updates"]
#sections = ["main"]
#mirror_source = false
#architectures = ["amd64"]

[mirror.security]
url = "http://security.debian.org/debian-security"
suites = ["buster/updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]

So with the security repo its run fine. In the buster, buster-updates repo go-apt-mirror stops with the error above. But only if i set the "source = true". When source = false, go-apt mirror download the 78Gb repo.
But the terminal output show me a lot of "missing file" like:

2020-04-16T19:54:41.910018Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/main/Contents-udeb-mips" repo="debian"
2020-04-16T19:54:41.911268Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/binary-mips/Packages" repo="debian"
2020-04-16T19:54:41.914372Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/Contents-udeb-i386" repo="debian"
2020-04-16T19:54:41.924035Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/debian-installer/binary-mips/Packages" repo="debian"

this output i get always when i start go-apt-mirror. What do this mean? This warning give me the feeling that anything isnt right. i am not sure.
I use the 1.4.1 release. This is my first issue. so i hope i make it right :-)

best regards
Chris

go-apt-mirror overlooks checksum mismatch between Release and InRelease

go-apt-mirror at its first stage downloads Release and InRelease indices.

Ref: https://wiki.debian.org/DebianRepository/Format#A.22Release.22_files

These should contain the same checksums for all other indices, but in reality
they may contain different checksums for some indices temporarily.

If Release and InRelease have different checksums for some indices,
current go-apt-mirror may overlook them and create incomplete mirror.

This bug may be fixed by detecting mismatches or by implementing
by-hash indices acquisition.

No binaries for 1.4.1 and 1.4.2

Readme says: "Pre-built binaries are available on releases.". But they are missing for 1.4.1 and 1.4.2.

In addition: This is software for caching/mirroring Debian packages. Shouldn't it be shipped as Debian packages itself (aka "Eat your own dogfood")?

[cacher] configure proxies via TOML

Currently, go-apt-cacher uses http.DefaultTransport which looks for proxy configurations from environment variables.

It would be more handy to be able to specify proxies in TOML.

Question: Have go-apt-cacher read go-apt-mirror files

I have successfully mirrored a Ubuntu repository, I'm not clear from the documentation if it is possible to have the go-apt-cacher mounted in such a way so that the cacher has the local mirrored repository available to it.

How is a client to be setup to use both? I guess you would change the source.list to point to the mirrored repository url that is being served with nginx? Then you would have the client http proxy setup to point to the go-apt-cacher url. How do you avoid the go-apt-cacher from caching the mirrored data?

Thanks for any clarification you can provide.