apt-mirror / apt-mirror Goto Github PK

View Code? Open in Web Editor NEW

413.0 30.0 230.0 150 KB

Official apt-mirror source.

Home Page: http://apt-mirror.github.com

License: GNU General Public License v2.0

Shell 3.05% Perl 94.57% Makefile 2.38%

apt-mirror's Introduction

apt-mirror

See: https://apt-mirror.github.io/

New maintainer(s) wanted

We (the current maintainers) lack the time and energy to maintain apt-mirror: Our last commit is years old and the number of pull request is rising. We agreed on acknowledging this fact and are searching for new maintainers who wants to join the GitHub apt-mirror group and continue maintaining this repository and do new releases. If you are interested and have time and energy to take the project over, please contact Brandon Holtsclaw to give you the permission.

apt-mirror's People

Contributors

Stargazers

Watchers

Forkers

joshdoe tyru rroemhild 805gi diosney guymatz dr4ke eifx cf-josecarlos carmenates09 ebrainte jwater7 pombredanne chrisportman th36r1m georgemarselis fbarbeira sunnynarula infobrand funkypopcorn jinwentan pfelecan nzrs tangfeixiong aalmenar greising rbracken pmhahn someuser89 djgalloway dmick ceph defnnn segmond esnellman connlan markthink cloud-devops mayanksuman chsk-code blablanet twang2218 sfinexer peterdavehello guoyu07 steveayre neatnerdprime chenjun3092 johncheung box293 yaunj codedad1983 imnx tknr kbdcalls efe13 josephharding larsjsol evil-talk evili quinndiggity lambda pbirokas wasapi nblock kjetilho jmccanta oieieio rinnaz duyguolmez pop-os josbraden sjtug pikachurus strausmann chrishagetlouts tanchoux shujiedev lostsquirrel r0mdau geosone francistheodorecatte dctucker fuyuanli liedekef ipwnosx holishing stifler6996 balapradeep tcboeira beberd alacn1 wxrui jntnl42 mattmattox hxun lostsnail42 univention sadpencil sdu-mirrors

apt-mirror's Issues

"InRelease" files aren't mirrored

Hello,

apt-mirror doesn't mirror "InRelease" files from repositories.

These files are mandatory for a network installation of Debian Wheezy.

Regards,

Unlink files instead of clobbering existing ones

We use apt-mirror for a local mirror. We want to keep some history for our mirrors, using hardlinks between two "snapshots".

The default behavior of wget is to clobber existing files if they changed. This is not something good for our setup since we want to be sure that NO file will change in previous snapshots.

We would like to use the wget option --unlink in apt-mirror. I'm working on a patch to enable a new variable for this.

apt-mirror behind university's firewall | Not stopping when quota exceeds

apt-mirror doesn't stop when it receives the university firewall's login page instead of the file it intends to download. This always happens when my quota for the day is over.

e.g., apt-repo/mirror/archive.ubuntu.com/ubuntu/dists/trusty/InRelease

<HTML><HEAD>
<meta http-equiv=pragma content=nocache>
<META HTTP-EQUIV=Expires CONTENT=-1>
<SCRIPT>
        location.href="http://10.1.2.3:80/httpclient.html"
</SCRIPT>
</HEAD><BODY>
</BODY>
</HTML>

This invariably corrupts the repository, making apt-mirror to deem most of the repository's files unnecessary,

Downloading 425 index files using 4 threads...
Begin time: Thu Sep  1 01:00:01 2016
[4]... [3]... [2]... [1]... [0]...
End time: Thu Sep  1 01:00:23 2016

Processing tranlation indexes: [TTTTTTTTTTTTTTT]

Downloading 0 translation files using 0 threads...
Begin time: Thu Sep  1 01:00:23 2016
[0]...
End time: Thu Sep  1 01:00:23 2016

Processing indexes: [SSSSSPPPPPPPPPPPPPPP]

0 bytes will be downloaded into archive.
Downloading 0 archive files using 0 threads...
Begin time: Thu Sep  1 01:00:24 2016
[0]...
End time: Thu Sep  1 01:00:24 2016

310.3 GiB in 294483 files and 29491 directories can be freed.
Run apt-repo/var/clean.sh for this purpose.

Running the Post Mirror script ...
(apt-repo/var/postmirror.sh)

Post Mirror script has completed. See above output for any possible errors.

Could you please suggest solutions to stop apt-mirror in such a scenario ?

debian 8 error

Fetched 437 kB in 1s (289 kB/s)
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-updates/main/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-updates/main/binary-i386/Packages Hash Sum mismatch

Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/source/Sources Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/source/Sources Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/source/Sources Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/binary-i386/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/binary-i386/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/binary-i386/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/source/Sources Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/non-free/source/Sources Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages Hash Sum mismatch

W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/binary-i386/Packages Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

And I delete the file will be ok!

rm -rf debian/dists/jessie-updates/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-updates/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/main/source/Sources.bz2
rm -rf debian/dists/jessie-backports/non-free/source/Sources.bz2
rm -rf debian/dists/jessie-backports/contrib/source/Sources.bz2
rm -rf debian/dists/jessie-backports/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/non-free/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/contrib/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/non-free/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/contrib/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/main/source/Sources.bz2
rm -rf debian/dists/jessie-proposed-updates/non-free/source/Sources.bz2
rm -rf debian/dists/jessie-proposed-updates/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-updates/main/binary-amd64/Packages
rm -rf debian/dists/jessie-updates/main/binary-i386/Packages
rm -rf debian/dists/jessie-backports/main/source/Sources
rm -rf debian/dists/jessie-backports/non-free/source/Sources
rm -rf debian/dists/jessie-backports/contrib/source/Sources
rm -rf debian/dists/jessie-backports/main/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/non-free/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/contrib/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/main/binary-i386/Packages
rm -rf debian/dists/jessie-backports/non-free/binary-i386/Packages
rm -rf debian/dists/jessie-backports/contrib/binary-i386/Packages
rm -rf debian/dists/jessie-proposed-updates/main/source/Sources
rm -rf debian/dists/jessie-proposed-updates/non-free/source/Sources
rm -rf debian/dists/jessie-proposed-updates/main/binary-amd64/Packages
rm -rf debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages
rm -rf debian/dists/jessie-proposed-updates/main/binary-i386/Packages

Autogenerated Packages.bz2 and Sources.bz2 yield a checksum mismatch

On ftp.at.debian.org only Packages.gz, Packages.xz as well as Sources.gz and Sources.xz are available for download [1,2].

Apt-mirror relies on the availability of Packages.bz2 and Sources.bz2. Since those do not exist on ftp.at.debian.org, apt-mirror seems to autocreate them. They appear on the filesystem with a timestamp matching the last apt-mirror run.

If a client issues an apt-get update, those generated files yield a Hash Sum mismatch error.

If Packages.bz2 or Sources.bz2 is not available on the server side, it should not be generated by apt-mirror. Furthermore, once Packages.bz2 or Sources.bz2 disappear on the server side at one point in time, it should also disappear on the mirror side.

affected version: 0.4.9-1

[1] ftp://ftp.at.debian.org/debian/dists/jessie/main/binary-amd64/
[2] ftp://ftp.at.debian.org/debian/dists/jessie/main/source/

Add Documentation

Add documentation that describes the configuration options and their default values.

( Mainly what is there does not cover the configuration options and what they do, only shows the default and one limited example. )

Add file integrity support

Add option to check md5 and sha signatures of files, like debmirror does.

Crash if too many repos

Hello,

If I add too many repos, more than 50, then apt-mirror crashes when downloading:

Processing indexes: [SSSSSSSSSSPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]

6.5 GiB will be downloaded into archive.
Downloading 931 archive files using 20 threads...
Use of uninitialized value $pid in numeric lt (<) at /usr/bin/apt-mirror line 252.
Use of uninitialized value $pid in numeric eq (==) at /usr/bin/apt-mirror line 254.

Thank you in advance for your help.

apt-mirror doesn't seem to know how to deal with ARM repos

A for instance is the Raspbian repo.

Processing indexes: [SPPPPPPPPPPPPPPPPsh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/main/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/contrib/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/non-free/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/rpi/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
]

I see that it is expecting "Packages.gz" when this repo contains "Contents-armhf.gz". Don't know if this is something that can be added easily or not but it would be great to have this is a functionality.

Missing "de_DE" in apt-mirror for downloading Translation

The following codesegment should be:

        my $lang;
        foreach $lang ( "ast", "bg", "bs", "ca", "cs", "da", "de", "de_DE", "el", "en", "en_AU", "en_CA", "en_GB", "eo", "es", "eu", "fa", "fi", "fr", "fr_FR", "gl", "hr", "hu", "id", "is", "it", "ja", "ka", "km", "ko", "lt", "ms", "my", "nb", "nl", "no_NB", "oc", "pl", "pt", "pt_BR", "ro", "ru", "si", "sk", "sl", "sq", "sr", "sv", "th", "tr", "uk", "vi", "zh_CN", "zh_HK", "zh_TW" )
        {
            add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang );
            add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang . ".gz" );
            add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang . ".bz2" );
        }
    add_url_to_download( $url . $_ . "/i18n/Index" );
    }

Batching of URLs is inefficient

If i'm not mistaken, apt-mirror works by splitting the files to be downloaded up into batches of files / nthreads files each. So if you've got 1000 files to download, and you've configured 2 threads, thread 0 gets 500 files and thread 1 gets 500 files.

The problem with this arrangement is that each set of files may be vastly different in size. For example, thread 0 might get 500 small command-line utilities and thread 1 might get 500 giant kernel images. So what ends up happening here is that all of your threads except one complete, and then you're left sitting there waiting for the last one to slowly pull down its massive pay-load all by itself. It gets especially bad if you have a per-thread download limit set — you could be waiting days for that one lonely instance of wget to finish.

The package lists contain the size of each of the packages, right? Would it be possible to change the apportioning of files so that each wget instance gets a roughly similar amount of data (rather than number of files) to fetch?

apt-mirror requires gzip compressed index files

apt-mirror will attempt to download both gzip and bzip2 compressed indexes but is hard coded to only attempt to parse indexes compressed with gzip. As a result, apt-mirror will fail to mirror a repository that does not publish gzip compressed indexes.

apt-mirror should attempt to download gzip, bzip2, lzma, and uncompressed indexes and be able to successfully mirror a repository if any one format is provided.

About Contribute

Hi @bholtsclaw

Thank you for your work behind apt-mirror maintenance, could you please provide a prioritized list of pending bugs/feat from people can start contributing?

Thank you.

Error: "Proceed indexes:" ... "gzip: stdin: not in gzip format"

[Problem partly solved, see next post. The failing download is not related to the error message.]

I am repeatedly updating the mirror, and getting exactly the same result every time (see below). It seems that there are some files to be downloaded, but they must be failing (?) which I'm guessing is related to the "gzip: stdin: not in gzip format" error. I cut the threads down from 20 to 5 to see if it would help, but the results do not change. (Same number of "PPP"s before/between/after the errors.)

I see that "apt-get update" seems to sometimes produce this error:
https://answers.launchpad.net/ubuntu/+question/20750

Also, this same error reported on the web in cases of a corrupted tar archive:
http://fogproject.org/forum/threads/gzip-stdin-not-in-gzip-format-error-on-ubuntu-server-12-04.791/

Here is one other apt-mirror report:
http://askubuntu.com/questions/283070/apt-mirror-error-qzip-stdin-not-in-gzip-format

Any idea what's happening/ how I can fix it (or even identify the problem files/repository)?

/DA

sudo -u apt-mirror apt-mirror
Downloading 502 index files using 5 threads...
Begin time: Thu Dec 19 23:32:43 2013
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Thu Dec 19 23:33:05 2013

Proceed indexes: [PPPPPPP
gzip: stdin: not in gzip format

gzip: stdin: not in gzip format
PPPPPPP
gzip: stdin: not in gzip format

gzip: stdin: not in gzip format
PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]

292.6 MiB will be downloaded into archive.
Downloading 16 archive files using 5 threads...
Begin time: Thu Dec 19 23:33:17 2013
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Thu Dec 19 23:33:18 2013

0.0 bytes in 0 files and 0 directories can be freed.
Run /media/MAXTOR-300G/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/media/MAXTOR-300G/apt-mirror/var/postmirror.sh)

Removing 0 unnecessary files [0 bytes]...
done.

Removing 0 unnecessary directories...
done.

Post Mirror script has completed. See above output for any possible errors.

Halting apt-mirror

When manually started from the terminal, it seems apt-mirror is easy to stop using ctrl+c.

However, if apt-mirror is started via crontab it must be terminated via:

pkill -x -f '/bin/sh -c /usr/bin/apt-mirror > /opt/apt-mirror/mirror/archive.ubuntu.com/ubuntu/apt-mirror.log' --signal HUP
kill $( pidof  wget )

or some other variant of these commands; since there is nothing to ctrl+c, there is no processes actually called apt-mirror, and because killing the apt-mirror process does not kill the 20 or so threads it created.

It would be nice if a feature were implement where one could just type:

apt-mirror stop

And halt the entire process.

Spaces in file paths?

I can't figure out how to specify a path with spaces in it (in mirror.list).
Enclosing in double quotes -> the double quotes are literally a part of the path.
Backslash before space -> the backslash becomes a part of the path.

Warnings on repositories without MD5sums

I'm using apt-mirror on Ubuntu 16.04, and some of the repositories I'm mirroring does not contain MD5sums in the Packages files. This results in warnings like this:

Global symbol "$package" requires explicit package name (did you forget to declare "my $package"?) at /usr/bin/apt-mirror line 787

Line 787 in Ubuntu 16.04 corresponds to line 798 in version 0.5.2.

print FILES_MD5 $lines{"MD5sum:"} . " " . remove_double_slashes( $path . "/" . $lines{"Filename:"} ) . "\n";

Appending if defined $lines{"MD5sum:"} to the line silences the error. However, you probably shouldn't silence all these possibly missing checksums. Maybe you could add some logic to warn if there isn't at least one checksum in the file?

There is no "debian-installer" sub-folder in "partner" repo

With apt-mirror from mpesari (which fixes #56) and an Ubuntu Xenial repo:

Processing indexes: [PPPPPPPPPPPPPsh: 1: cannot open archive.canonical.com/ubuntu//dists/xenial/partner/debian-installer/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 773.
Psh: 1: cannot open archive.canonical.com/ubuntu//dists/xenial/partner/debian-installer/binary-i386/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 773.
]

In /etc/apt/mirror.list:

deb      http://archive.canonical.com/ubuntu xenial partner
deb-i386 http://archive.canonical.com/ubuntu xenial partner
clean http://archive.canonical.com

Use of uninitialized value $config{"options"}

Multiple warnings about uninitialized value $config{"options"}

apt-mirror_err.txt
perl_ver.txt

Add support for leightweight mirroring to S3?

Hi,
we'd like to have support for lightweight mirroring to S3. As far as I can see now, this would entail pulling a list of packages from an S3 bucket (the target) and then pulling a list of packages from Upstream, merging both lists and dumping everything to S3 again.

Would this change be in scope for your project?

Cheers, Mark

use of skip-clean by distribution

I mirror different Debian versions, and would like to keep the old files for only the last distribution. I think that's the purpose of skip-clean directive, but can't get it to work.

I tested with differents line formats :

clean http://ftp.fr.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.fr.debian.org/debian/dists/wheezy/

OR
clean http://ftp.fr.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.fr.debian.org/debian wheezy main contrib non-free
skip-clean http://security.debian.org/ wheezy/updates main contrib non-free
skip-clean http://ftp.fr.debian.org/debian/ wheezy-updates main contrib non-free

In the 2 cases, the whole "ftp.fr.debian.org/debian" is skipped.

Is there anything that I miss ? Or is it possible to do what I want with clean/skip-clean options ?

Cheers

not download by-hash from source

I just build new mirror for Ubuntu 16.04 Xenial and apt-mirror won't download the by-hash directory at all.

For example :
http://re.archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/by-hash/

My local mirror after update it shows "404 not found"

Thx for your support

Add aria2 or axel as optional downloader.

tl;dr

The backend downloader for apt-mirror is wget. Does it make sense to add support for aria2 or axel as optional downloaders?

axel is more light-weight and has less dependencies.
aria2 is heavier with more features that may or may not be helpful in the case of apt-mirror.

wget is a single-threaded downloader, though apt-mirror has a lot of code to spawn n-threads of wget. From what I can read in the code, it pulls down the indexes and then splits up the to-be-downloaded URIs into archive-urls.n which are fed to individual wget processes. Not bad, but it only helps where each connection to the server is throttled. It does nothing towards pulling the same file with multiple connections to one server, or to pull from multiple servers. The plus side of using wget is that it is installed by default on most Linux servers and is a well-known tool.

aria2 or axel, on the other hand, are multi-threaded downloaders to begin with. All the logic of multi-threading the download list can be handled by one of these instead of apt-mirror. Due to wget's ubiquity, you aren't going trash that code, but I imagine that adding support for aria2 or axel should be fairly straight-forward.

The main benefit is that these downloaders can pull the same file from multiple servers or the same file with multiple connections. apt-mirror with wget cannot do either. This would be a substantial benefit where we use apt-mirror with a really low-end connection.

Glad to see apt-mirror getting some polish!

apt-mirror warning after change deb source

OS: ubuntu16.04(apt-mirror default version 0.5.3.1)
apt-get remove apt-mirror && apt-get purge apt-mirror
I download latest version(0.5.4) from github and make install
this is my mirror.list

set base_path         /var/spool/apt-mirror
set mirror_path       $base_path/mirror
set skel_path         $base_path/skel
set var_path          $base_path/var
set postmirror_script $var_path/postmirror.sh
set defaultarch       amd64
set run_postmirror    0
set nthreads          20
set limit_rate        100m
set _tilde            0
# Use --unlink with wget (for use with hardlinked directories)
set unlink            1
set use_proxy         off
set http_proxy        127.0.0.1:3128
set proxy_user        user
set proxy_password    password


deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main
# deb-src http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main
clean http://ubuntu-cloud.archive.canonical.com/ubuntu

then I run apt-mirror
output:

root@dev:~# apt-mirror
Downloading 14 index files using 14 threads...
Begin time: Mon Aug 28 11:13:31 2017
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Mon Aug 28 11:13:31 2017

Processing translation indexes: [T]

Downloading 0 translation files using 0 threads...
Begin time: Mon Aug 28 11:13:31 2017
[0]... 
End time: Mon Aug 28 11:13:31 2017

Processing DEP-11 indexes: [D]

Downloading 0 dep11 files using 0 threads...
Begin time: Mon Aug 28 11:13:31 2017
[0]... 
End time: Mon Aug 28 11:13:31 2017

Processing indexes: [P]

242.4 MiB will be downloaded into archive.
Downloading 735 archive files using 20 threads...
Begin time: Mon Aug 28 11:13:31 2017
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Mon Aug 28 11:13:32 2017

0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

root@dev:~#

It seems OK, but when I uncomment the deb-src line and run apt-mirror
Output:

root@dev:~# apt-mirror
Downloading 52 index files using 20 threads...
Begin time: Mon Aug 28 11:21:43 2017
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Mon Aug 28 11:21:43 2017

Processing translation indexes: [TTTT]

Downloading 0 translation files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]... 
End time: Mon Aug 28 11:21:43 2017

Processing DEP-11 indexes: [DDDD]

Downloading 0 dep11 files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]... 
End time: Mon Aug 28 11:21:43 2017

apt-mirror: can't open index ubuntu-cloud.archive.canonical.com/ubuntu//dists/xenial-updates/ocata/main/source/Sources in process_index at /usr/local/bin/apt-mirror line 800, <STREAM> chunk 872.
Processing indexes: [SPPPP]

0 bytes will be downloaded into archive.
Downloading 0 archive files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]... 
End time: Mon Aug 28 11:21:43 2017

0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

root@dev:~#

I wander if there is any problem

Cannot handle 'deb [arch=amd64] http://...' format

When I try to create a mirror of cloudera-cdh5, I found this error.

The apt list file is under /etc/apt/sources.list.d/cloudera-cdh5.list on my system, and the content is :

deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
deb-src http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib

I try to do the mirror by following command and got errors:

vagrant@ubuntu:~$ sudo apt-mirror /etc/apt/sources.list.d/cloudera-cdh5.list 
Downloading 15 index files using 15 threads...
Begin time: Mon Apr 14 17:04:35 2014
[15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... 
End time: Mon Apr 14 17:04:36 2014

Proceed indexes: [SPsh: 1: cannot open [arch=amd64]//dists/http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/precise-cdh5/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in proceed_index_gz at /usr/bin/apt-mirror line 460.
vagrant@ubuntu:~$

If I have to change the apt .list to the form below to make it work:

deb-amd64 http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
deb-src http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib

OS: Ubuntu 12.04
apt-mirror: 0.4.8-5

Could you add support of [arch=amd64]-like format? Thanks.

exclude dirs from clean.sh

i would love to see an exclude dirs in config for clean.sh
coz when i pull the installer-amd64 / i386 directorys via postmirror.sh script
i don´t want to redownload them each time clean.sh runs ,
or how about adding them to apt-mirror

great work btw.

thx

Merge refactoring from kimmel?

Of all the forks I've found, @kimmel has the most extensive changes, primarily in the form of refactoring, cleanup, and conformance to perlcritic:
https://github.com/kimmel/apt-mirror-fork

I've taken a quick look, but not long enough to know if the changes should be merged here.

Dependency packages - not installing?

I am finding that on my Ubuntu 12.04, certain dependency packages are listed as available in the update manager, but don't update because (it says) they can't be downloaded:

linux-generic
linux-headers-generic
linux-image-generic

The description of that first package, for example, is:

"This package will always depend on the latest complete generic Linux kernel available."

What's the story here? Is there something that can be done about this?

/DA

Contents of the associated directory for linux-generic:

mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal-eol-upgrade_3.5.0.44.50_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring-eol-upgrade_3.8.0.34.34_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.23.25_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.23.25_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.23.25_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.57.68_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal-eol-upgrade_3.5.0.44.50_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring-eol-upgrade_3.8.0.34.34_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.57.68_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_amd64.deb

patch for Contents

I'm using apt-mirror for Debian repositories mirroring and even contributed a packaged version for Solaris...

In certain circumstances, such as when using live-build, the file Contents is required in some nodes of the repository tree, e.g. debian/dists/testing/main/.

Even though there is some processing related to this kind of file, it's not complete. That is the reason for which I propose a patch to completely acquire the required Contents file and, for completeness sake, I added also the acquisition of the InRelease file.

Thank you in advance for considering this patch

Note that I already contacted Brandon on this issue and he suggested to create a pull. However, I didn't succeeded to create a branch or I don't really understand how to create a pull. This is why I put the patch here:

*** /usr/bin/apt-mirror 2013-03-03 06:06:55.000000000 +0100
--- /home/peter/org/usr/bin/apt-mirror  2014-01-27 13:54:29.000000000 +0100
***************
*** 362,371 ****
      $url = $uri . "/dists/" . $distribution . "/";

      add_url_to_download($url . "Release");
      add_url_to_download($url . "Release.gpg");
!         if (get_variable("_contents")) {
!             add_url_to_download($url . "Contents-" . $arch . ".gz");
!             add_url_to_download($url . "Contents-" . $arch . ".bz2");
!         }
      foreach (@components) {
          add_url_to_download($url . $_ . "/binary-" . $arch . "/Release");
--- 362,376 ----
      $url = $uri . "/dists/" . $distribution . "/";

+     add_url_to_download($url . "InRelease");
      add_url_to_download($url . "Release");
      add_url_to_download($url . "Release.gpg");
! 
!   if (get_variable("_contents")) {
!       foreach(@components)
!       {
!           add_url_to_download($url . $_ . "/Contents-" . $arch . ".gz");
!           add_url_to_download($url . $_ . "/Contents-" . $arch . ".bz2");
!       }
!   }
      foreach (@components) {
          add_url_to_download($url . $_ . "/binary-" . $arch . "/Release");

Add in-config proxy support

Add option to config file to configure proxies.

This proxy can be parsed and added to low-level wget commands as an option.

not download DEP-11 from source

I just build new mirror for Ubuntu 16.04 Xenial and apt-mirror won't download the dep11 directory at all.

The source has:
http://ca.archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/

My local mirror after update it shows "404 not found"

how to set defaultarch in mirror.list for both i386 and amd64

I can not find any doc or examples to tell how to configurate the defaultarch in mirror.list
I want to mirror both i386 and amd64
I just do the follow steps
set defaultarch i386 and run sudo apt-mirror
set defaultarch amd64 and run sudo apt-mirror
:(

Pass config variables to postmirror.sh

I'm doing some 'advanced' items in my postmirror.sh and I need access to the config variables from apt-mirror. Specifically in my case base_path, mirror_path, & var_path. Currently I added the following lines to apt-mirror for the variables to be in the environment for postmirror.sh:

$ENV{base_path}=get_variable( "base_path" );
$ENV{mirror_path}=get_variable( "mirror_path" );
$ENV{var_path}=get_variable( "var_path" );

But this may be better handled by something like the following code? (I'm not a perl programmer)

######################################################################################
## Setting Environment variables to be used post mirror

foreach my $key ( keys %config_variables )
{
    $ENV{$key}=get_variable( "$key" );
}

Or is there a better way to get access to these variables from inside the postmirror.sh script?

Configurable apt-mirror base directory.

Hello, I need to be able to define what the apt-mirror base directory is via an argument.

Currently, it's hard-coded to /var/spool/apt-mirror but I need to use different directories instead to ease technical complications caused by the hard-coded path.

Thanks,
Wayne

Add information about mirroring progress

When downloading add the following information to the output:

Progress in % (fi: 56%) (Archive files already downloaded / Archive files remaining) * 100
Estimated remaining time.

Thanks :D

shouldn't apt-mirror not overwrite index files that haven't changed?

i note this code in copy_file()

if ( get_variable("unlink") == 1 )

{

    if ( compare( $from, $to ) != 0 ) { unlink($to); }

}

unless ( copy( $from, $to ) )

{

    warn("apt-mirror: can't copy $from to $to");

    return;

}

checks to see if the files are the same before unlinking, but then it goes ahead and tries to rewrite the file in any case. i can't figure out what sense that makes: if you are always going to overwrite the file, then why bother doing the comparison before unlinking, if unlink=1? why not just stat the $to file and unlink it if it exists instead?

(interesting, but totally beside the point: if the $to file doesn't exist, compare() should return -1 and so the code willl try unlinking the nonexistent file, if i read the perldoc right. for that matter, you could probably dispense with even trying to stat() and just unlink() away, since there's no check for an error, so "no such file" shouldn't hurt)

not trying to criticize, since i can't even begin to approach the awesomeness that went into creating this script. only reason i am bringing this up is because i get a permissions failure here when copy() tries to overwrite the file that didn't get unlink()-ed because compare() == 0. better to skip that check, for me. i actually just rewrote that bit to always do the comparison and to skip the overwriting, if the files compare equal, whether any unlinking is to be done or not (although this left me with one more place where the script still failed on me: utime() failed on a file i didn't have ownership of). anyways, i've attached my patch, if anybody has use for it. (change the relevant die, from the original script, to warn if you also don't care about utime() failing, though i'm not including that in the attached, since that maybe makes the script unreliable for mirroring archives where timestamp fidelity is critical, like for rsync-ing or what have you. seems to me that apt just cares about checksums, but i may end up finding out the hard way if not. also, anybody who knows Perl, please fix the bugs and stylistic infelicities and share, so i can update my copy, thanks. [note that i left some whitespace glitches in place to minimize the patch. feel free to re-indent])

Update old apt-mirror with updated apt-mirror

I'll like to know is it's possible to update an older (apt-)mirror on a local server using a recent one witout creating the whole new tree structure and copy a second repository right next to the old one.

Is that possible, is it DNS related?

Launchpad Repos All Correctly Configured & Downloading - but members of 2 won't install.

I've got a number of Launchpad repos, all of which are working but two. These two appear to be correctly configured and are downloading correctly. (mirrored contents, below.) Nevertheless, while the update manager in ubuntu lists the packages, it says they can't be downloaded. Any thoughts on how to approach this?

Other updates (LP-PPA-b-eltzner-qpdfview)
tabbed document viewer
qpdfview (Size: 360 kB)

tabbed document viewer - DjVu plugin
qpdfview-djvu-plugin (Size: 35 kB)

tabbed document viewer - PostScript plugin
qpdfview-ps-plugin (Size: 29 kB)

and:

Other updates (LP-PPA-nemh-gambas3)
Gambas native form component
gambas3-gb-form (Size: 260 kB)

Gambas utilities class
gambas3-gb-settings (Size: 12 kB)

Gambas runtime interpreter
gambas3-runtime (Size: 363 kB)

/DA

qpdfview:
http://ppa.launchpad.net/b-eltzner/qpdfview/ubuntu/

qpdfview_0.4.7-1precise1_amd64.deb
qpdfview_0.4.7-1precise1_i386.deb
qpdfview-djvu-plugin_0.4.7-1precise1_amd64.deb
qpdfview-djvu-plugin_0.4.7-1precise1_i386.deb
qpdfview-languages_0.4.5-1precise3_amd64.deb
qpdfview-languages_0.4.5-1precise3_i386.deb
qpdfview-ps-plugin_0.4.7-1precise1_amd64.deb
qpdfview-ps-plugin_0.4.7-1precise1_i386.deb
qpdfview-translations_0.4.7-1precise1_amd64.deb
qpdfview-translations_0.4.7-1precise1_i386.deb

gambas3:
http://ppa.launchpad.net/nemh/gambas3/ubuntu/pool/main/g/gambas3/

gambas3_3.5.1-0precise2_all.deb
gambas3-dev_3.5.1-0precise2_amd64.deb
gambas3-dev_3.5.1-0precise2_i386.deb
gambas3-examples_3.5.1-0precise2_all.deb
gambas3-gb-args_3.5.1-0precise2_all.deb
gambas3-gb-cairo_3.5.1-0precise2_amd64.deb
gambas3-gb-cairo_3.5.1-0precise2_i386.deb
gambas3-gb-chart_3.5.1-0precise2_all.deb
gambas3-gb-clipper_3.5.1-0precise2_amd64.deb
gambas3-gb-clipper_3.5.1-0precise2_i386.deb
gambas3-gb-complex_3.5.1-0precise2_amd64.deb
gambas3-gb-complex_3.5.1-0precise2_i386.deb
gambas3-gb-compress_3.5.1-0precise2_amd64.deb
gambas3-gb-compress_3.5.1-0precise2_i386.deb
gambas3-gb-compress-bzlib2_3.5.1-0precise2_amd64.deb
gambas3-gb-compress-bzlib2_3.5.1-0precise2_i386.deb
gambas3-gb-compress-zlib_3.5.1-0precise2_amd64.deb
gambas3-gb-compress-zlib_3.5.1-0precise2_i386.deb
gambas3-gb-crypt_3.5.1-0precise2_amd64.deb
gambas3-gb-crypt_3.5.1-0precise2_i386.deb
gambas3-gb-data_3.5.1-0precise2_amd64.deb
gambas3-gb-data_3.5.1-0precise2_i386.deb
gambas3-gb-db_3.5.1-0precise2_amd64.deb
gambas3-gb-db_3.5.1-0precise2_i386.deb
gambas3-gb-db-form_3.5.1-0precise2_all.deb
gambas3-gb-db-mysql_3.5.1-0precise2_amd64.deb
gambas3-gb-db-mysql_3.5.1-0precise2_i386.deb
gambas3-gb-db-odbc_3.5.1-0precise2_amd64.deb
gambas3-gb-db-odbc_3.5.1-0precise2_i386.deb
gambas3-gb-db-postgresql_3.5.1-0precise2_amd64.deb
gambas3-gb-db-postgresql_3.5.1-0precise2_i386.deb
gambas3-gb-db-sqlite2_3.5.1-0precise2_amd64.deb
gambas3-gb-db-sqlite2_3.5.1-0precise2_i386.deb
gambas3-gb-db-sqlite3_3.5.1-0precise2_amd64.deb
gambas3-gb-db-sqlite3_3.5.1-0precise2_i386.deb
gambas3-gb-dbus_3.5.1-0precise2_amd64.deb
gambas3-gb-dbus_3.5.1-0precise2_i386.deb
gambas3-gb-desktop_3.5.1-0precise2_amd64.deb
gambas3-gb-desktop_3.5.1-0precise2_i386.deb
gambas3-gb-desktop-gnome_3.5.1-0precise2_amd64.deb
gambas3-gb-desktop-gnome_3.5.1-0precise2_i386.deb
gambas3-gb-eval-highlight_3.5.1-0precise2_all.deb
gambas3-gb-form_3.5.1-0precise2_all.deb
gambas3-gb-form-dialog_3.5.1-0precise2_all.deb
gambas3-gb-form-mdi_3.5.1-0precise2_all.deb
gambas3-gb-form-stock_3.5.1-0precise2_all.deb
gambas3-gb-gmp_3.5.1-0precise2_amd64.deb
gambas3-gb-gmp_3.5.1-0precise2_i386.deb
gambas3-gb-gsl_3.5.1-0precise2_amd64.deb
gambas3-gb-gsl_3.5.1-0precise2_i386.deb
gambas3-gb-gtk_3.5.1-0precise2_amd64.deb
gambas3-gb-gtk_3.5.1-0precise2_i386.deb
gambas3-gb-gtk-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-gtk-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-httpd_3.5.1-0precise2_amd64.deb
gambas3-gb-httpd_3.5.1-0precise2_i386.deb
gambas3-gb-image_3.5.1-0precise2_amd64.deb
gambas3-gb-image_3.5.1-0precise2_i386.deb
gambas3-gb-image-effect_3.5.1-0precise2_amd64.deb
gambas3-gb-image-effect_3.5.1-0precise2_i386.deb
gambas3-gb-image-imlib_3.5.1-0precise2_amd64.deb
gambas3-gb-image-imlib_3.5.1-0precise2_i386.deb
gambas3-gb-image-io_3.5.1-0precise2_amd64.deb
gambas3-gb-image-io_3.5.1-0precise2_i386.deb
gambas3-gb-jit_3.5.1-0precise2_amd64.deb
gambas3-gb-jit_3.5.1-0precise2_i386.deb
gambas3-gb-libxml_3.5.1-0precise2_amd64.deb
gambas3-gb-libxml_3.5.1-0precise2_i386.deb
gambas3-gb-logging_3.5.1-0precise2_all.deb
gambas3-gb-map_3.5.1-0precise2_all.deb
gambas3-gb-media_3.5.1-0precise2_amd64.deb
gambas3-gb-media_3.5.1-0precise2_i386.deb
gambas3-gb-memcached_3.5.1-0precise2_all.deb
gambas3-gb-mime_3.5.1-0precise2_amd64.deb
gambas3-gb-mime_3.5.1-0precise2_i386.deb
gambas3-gb-mysql_3.5.1-0precise2_amd64.deb
gambas3-gb-mysql_3.5.1-0precise2_i386.deb
gambas3-gb-ncurses_3.5.1-0precise2_amd64.deb
gambas3-gb-ncurses_3.5.1-0precise2_i386.deb
gambas3-gb-net_3.5.1-0precise2_amd64.deb
gambas3-gb-net_3.5.1-0precise2_i386.deb
gambas3-gb-net-curl_3.5.1-0precise2_amd64.deb
gambas3-gb-net-curl_3.5.1-0precise2_i386.deb
gambas3-gb-net-pop3_3.5.1-0precise2_amd64.deb
gambas3-gb-net-pop3_3.5.1-0precise2_i386.deb
gambas3-gb-net-smtp_3.5.1-0precise2_amd64.deb
gambas3-gb-net-smtp_3.5.1-0precise2_i386.deb
gambas3-gb-openal_3.5.1-0precise2_amd64.deb
gambas3-gb-openal_3.5.1-0precise2_i386.deb
gambas3-gb-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-glsl_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-glsl_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-glu_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-glu_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-sge_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-sge_3.5.1-0precise2_i386.deb
gambas3-gb-openssl_3.5.1-0precise2_amd64.deb
gambas3-gb-openssl_3.5.1-0precise2_i386.deb
gambas3-gb-option_3.5.1-0precise2_amd64.deb
gambas3-gb-option_3.5.1-0precise2_i386.deb
gambas3-gb-pcre_3.5.1-0precise2_amd64.deb
gambas3-gb-pcre_3.5.1-0precise2_i386.deb
gambas3-gb-pdf_3.5.1-0precise2_amd64.deb
gambas3-gb-pdf_3.5.1-0precise2_i386.deb
gambas3-gb-qt4_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-ext_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-ext_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-webkit_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-webkit_3.5.1-0precise2_i386.deb
gambas3-gb-report_3.5.1-0precise2_all.deb
gambas3-gb-sdl_3.5.1-0precise2_amd64.deb
gambas3-gb-sdl_3.5.1-0precise2_i386.deb
gambas3-gb-sdl-sound_3.5.1-0precise2_amd64.deb
gambas3-gb-sdl-sound_3.5.1-0precise2_i386.deb
gambas3-gb-settings_3.5.1-0precise2_all.deb
gambas3-gb-signal_3.5.1-0precise2_amd64.deb
gambas3-gb-signal_3.5.1-0precise2_i386.deb
gambas3-gb-v4l_3.5.1-0precise2_amd64.deb
gambas3-gb-v4l_3.5.1-0precise2_i386.deb
gambas3-gb-vb_3.5.1-0precise2_amd64.deb
gambas3-gb-vb_3.5.1-0precise2_i386.deb
gambas3-gb-web_3.5.1-0precise2_all.deb
gambas3-gb-xml_3.5.1-0precise2_amd64.deb
gambas3-gb-xml_3.5.1-0precise2_i386.deb
gambas3-gb-xml-html_3.5.1-0precise2_amd64.deb
gambas3-gb-xml-html_3.5.1-0precise2_i386.deb
gambas3-gb-xml-rpc_3.5.1-0precise2_all.deb
gambas3-gb-xml-xslt_3.5.1-0precise2_amd64.deb
gambas3-gb-xml-xslt_3.5.1-0precise2_i386.deb
gambas3-ide_3.5.1-0precise2_all.deb
gambas3-runtime_3.5.1-0precise2_amd64.deb
gambas3-runtime_3.5.1-0precise2_i386.deb
gambas3-scripter_3.5.1-0precise2_all.deb

mirror.list is not documented

The mirror.list configuration supports many options, the file is well commented explaining each option.

mirror.list contains a single explanatory comment. The manpage is, at best, severly overstating how "well commented" mirror.list is.

Support For SSL based repo's with keys

Would it be possible to support repos that use SSL keys to authenticate

nginx repo comes to mind.

Add arch specific armhf

When I try to mirror this specific architecture debian repo, it fails because apt-mirror recognise deb-armhf as deb-arm.
Why armhf is not available on arch specific support?

apt-mirror BUG

     exec 'wget', '--no-cache', '--limit-rate=' . get_variable("limit_rate"), '-t', '5', '-r', '-N', '-c', '-l', 'inf', '-o', get_variable("var_path") . "/$stage-log.$i", '-i', get_variable("var_path") . "/$stage-urls.$i",

ADD OPTION '-c'
At a stop, there is no continuation or broken download resume.
The file remains not finished!

Components-$arch.yml.xz files not picked up by apt-mirror

Hi,

these files are listed in the Release files but ignored by apt-mirror. I parse the Release files and do a md5sum-check on the files listed there, leading to this error:

md5sum: main/dep11/Components-amd64.yml.xz: No such file or directory

Here is one example in the ubuntu repos: http://de.archive.ubuntu.com/ubuntu/dists/devel/main/dep11/Components-amd64.yml.xz

Checksums of downloaded files

Firstly I know that apt checks the signature before installing, that isn't the point of this issue - read on.

Apt-mirror tool doesn't seem to check the checksum of the downloaded file vs the checksum of the file as is reported in the Release file. Nor does it check the checksum of the previously downloaded file on disk vs the file on the remote server to ensure the local version isn't corrupt.

I believe this is necessary in order to ensure that you have mirrored a valid repository and that failures will not occour later when you attempt to install the packages. Also in case of local filesystem corruption you are able to repair the repository without needing to download everything again.

It is evident that this was intended at one point but does not appear to be fully implemented.

typo - tranlation

when I run the command:

apt-mirror

one of the messages is:

Processing tranlation indexes: [T]

but it seems that the word 'tranlation' should be 'translation'

I'm using apt-mirror in a debian testing. version:

$ dpkg -l apt-mirror
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Nome Versão Arquitectura Descrição
+++-==============-============-============-==================================
ii apt-mirror 0.5.1-1 all APT sources mirroring tool

Old InRelease file left

Hi,,

We've noticed that there is no InRelease file for Debian 8.2 in

ftp://ftp.se.debian.org/debian/dists/jessie/

but there was such file for Debian 8.1 in this dir and apt-mirror seems to skip cleaning InRelease files; skipping code in apt-mirror (*):

foreach ( keys %urls_to_download )
{
s[^(\w+)://][];
s[~][%7E]g if get_variable("tilde");
$skipclean{$} = 1;
$skipclean{$} = 1 if s[.gz$][];
$skipclean{$} = 1 if s[.bz2$][];
$skipclean{$_} = 1 if s[.xz$][];
}

The result of mirroring

ftp://ftp.se.debian.org/debian/dists/jessie/

with apt-mirror has now old InRelease (left from Debian 8.1) and new Release and Release.gpg (from Debian 8.2) which generates apt-get Hash Sum errors because apt-get seems to process InRelease if found, not Release+Release.gpg.

Also, the code (*) seems to have a bug: line with "$skipclean{$_} = 1;" already marks all the files from urls_to_download probably and lines

$skipclean{$_} = 1 if s[\.gz$][];
$skipclean{$_} = 1 if s[\.bz2$][];
$skipclean{$_} = 1 if s[\.xz$][];

do nothing (maybe the intention was to set it to 0 there to allow cleaning compressed stuff if compression format changes)?

What is the whole urls_to_download cleaning skipping for? What if repo maintainers change compression format from bz2 to xz or switch from InRelease to Release+Release.gpg only or from Release+Release.gpg to InRelease only?

Is it a problem to remove whole (*) code and let apt-mirror to sync all files from urls_to_download 1:1 without leaving old stuff there?

Regards,
Pawel

sanitise_url is not correctly picking up specified ports

The regex is incorrect:

Change: $uri =~ s&:\d+/&/&; # and port information
To: $uri =~ s&:\d+/$&/&; # and port information

_tilde option undocumented

There is a value called _tilde that is set to 0 in the config file (mirror.list). There is however no mentioning of it in the man page nor in the code commentary. It would be useful to know what the value does/change. A short mention in the man page or the README.md would not go amiss.

Always error with both i386 and amd64 arch

If i have both deb-i386 http:// and deb-amd64 http:// (same site) in /etc/apt/mirror.list, apt-mirror always raises error like below. But it works fine if i have only one arch (e.g. deb http://...).

apt-mirror: can't open index <site>/debian//dists/jessie/jessie/binary-i386/Packages in process_index at /usr/bin/apt-mirror line 800, <STREAM> chunk 576.
Processing indexes: [PPapt-mirror: can't open index <site>/debian//dists/stretch/stretch/binary-i386/Packages in process_index at /usr/bin/apt-mirror line 800.
PPPPPP]

My /etc/apt/mirror.list:

############# config ##################
set base_path /opt/apt-mirror

# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
set run_postmirror 0

set nthreads     10
set _tilde 0
############# end config ##############

# Debian 8
deb-i386 https://<site>/debian jessie jessie
deb https://<site>/debian jessie jessie

# Debian 9
deb-i386 https://<site>/debian stretch stretch
deb https://<site>/debian stretch stretch

# Ubuntu 14.04 LTS
deb-i386 https://<site>/ubuntu trusty trusty
deb https://<site>/ubuntu trusty trusty

# Ubuntu 16.04 LTS
deb-i386 https://<site>/ubuntu xenial xenial
deb https://<site>/ubuntu xenial xenial

apt-mirror does not download all Packages.XYZ and Sources.XYZ variants

apt-mirror currently downloads only the following Packages files in dists/<distribution>/<component>/binary-<arch>/:

Packages.gz
Packages.bz2

and the following Sources files in dists/<distribution>/<component>/source/:

Sources.gz
Sources.bz2

At least on ftp.at.debian.org, Packages.xz and Sources.xz are available too – apt-mirror should fetch those.

affected version: 0.4.9-1

apt-mirror / apt-mirror Goto Github PK

apt-mirror's Introduction

apt-mirror

New maintainer(s) wanted

apt-mirror's People

Contributors

Stargazers

Watchers

Forkers

apt-mirror's Issues

tl;dr

More

Recommend Projects

Recommend Topics

Recommend Org