apt-mirror / apt-mirror Goto Github PK
View Code? Open in Web Editor NEWOfficial apt-mirror source.
Home Page: http://apt-mirror.github.com
License: GNU General Public License v2.0
Official apt-mirror source.
Home Page: http://apt-mirror.github.com
License: GNU General Public License v2.0
I've got a number of Launchpad repos, all of which are working but two. These two appear to be correctly configured and are downloading correctly. (mirrored contents, below.) Nevertheless, while the update manager in ubuntu lists the packages, it says they can't be downloaded. Any thoughts on how to approach this?
Other updates (LP-PPA-b-eltzner-qpdfview)
tabbed document viewer
qpdfview (Size: 360 kB)
tabbed document viewer - DjVu plugin
qpdfview-djvu-plugin (Size: 35 kB)
tabbed document viewer - PostScript plugin
qpdfview-ps-plugin (Size: 29 kB)
and:
Other updates (LP-PPA-nemh-gambas3)
Gambas native form component
gambas3-gb-form (Size: 260 kB)
Gambas utilities class
gambas3-gb-settings (Size: 12 kB)
Gambas runtime interpreter
gambas3-runtime (Size: 363 kB)
/DA
qpdfview:
http://ppa.launchpad.net/b-eltzner/qpdfview/ubuntu/
qpdfview_0.4.7-1precise1_amd64.debprecise1_i386.deb
qpdfview_0.4.7-1
qpdfview-djvu-plugin_0.4.7-1precise1_amd64.debprecise1_i386.deb
qpdfview-djvu-plugin_0.4.7-1
qpdfview-languages_0.4.5-1precise3_amd64.debprecise3_i386.deb
qpdfview-languages_0.4.5-1
qpdfview-ps-plugin_0.4.7-1precise1_amd64.debprecise1_i386.deb
qpdfview-ps-plugin_0.4.7-1
qpdfview-translations_0.4.7-1precise1_amd64.debprecise1_i386.deb
qpdfview-translations_0.4.7-1
gambas3:
http://ppa.launchpad.net/nemh/gambas3/ubuntu/pool/main/g/gambas3/
gambas3_3.5.1-0precise2_all.deb
gambas3-dev_3.5.1-0precise2_amd64.deb
gambas3-dev_3.5.1-0precise2_i386.deb
gambas3-examples_3.5.1-0precise2_all.deb
gambas3-gb-args_3.5.1-0precise2_all.deb
gambas3-gb-cairo_3.5.1-0precise2_amd64.deb
gambas3-gb-cairo_3.5.1-0precise2_i386.deb
gambas3-gb-chart_3.5.1-0precise2_all.deb
gambas3-gb-clipper_3.5.1-0precise2_amd64.deb
gambas3-gb-clipper_3.5.1-0precise2_i386.deb
gambas3-gb-complex_3.5.1-0precise2_amd64.deb
gambas3-gb-complex_3.5.1-0precise2_i386.deb
gambas3-gb-compress_3.5.1-0precise2_amd64.deb
gambas3-gb-compress_3.5.1-0precise2_i386.deb
gambas3-gb-compress-bzlib2_3.5.1-0precise2_amd64.deb
gambas3-gb-compress-bzlib2_3.5.1-0precise2_i386.deb
gambas3-gb-compress-zlib_3.5.1-0precise2_amd64.deb
gambas3-gb-compress-zlib_3.5.1-0precise2_i386.deb
gambas3-gb-crypt_3.5.1-0precise2_amd64.deb
gambas3-gb-crypt_3.5.1-0precise2_i386.deb
gambas3-gb-data_3.5.1-0precise2_amd64.deb
gambas3-gb-data_3.5.1-0precise2_i386.deb
gambas3-gb-db_3.5.1-0precise2_amd64.deb
gambas3-gb-db_3.5.1-0precise2_i386.deb
gambas3-gb-db-form_3.5.1-0precise2_all.deb
gambas3-gb-db-mysql_3.5.1-0precise2_amd64.deb
gambas3-gb-db-mysql_3.5.1-0precise2_i386.deb
gambas3-gb-db-odbc_3.5.1-0precise2_amd64.deb
gambas3-gb-db-odbc_3.5.1-0precise2_i386.deb
gambas3-gb-db-postgresql_3.5.1-0precise2_amd64.deb
gambas3-gb-db-postgresql_3.5.1-0precise2_i386.deb
gambas3-gb-db-sqlite2_3.5.1-0precise2_amd64.deb
gambas3-gb-db-sqlite2_3.5.1-0precise2_i386.deb
gambas3-gb-db-sqlite3_3.5.1-0precise2_amd64.deb
gambas3-gb-db-sqlite3_3.5.1-0precise2_i386.deb
gambas3-gb-dbus_3.5.1-0precise2_amd64.deb
gambas3-gb-dbus_3.5.1-0precise2_i386.deb
gambas3-gb-desktop_3.5.1-0precise2_amd64.deb
gambas3-gb-desktop_3.5.1-0precise2_i386.deb
gambas3-gb-desktop-gnome_3.5.1-0precise2_amd64.deb
gambas3-gb-desktop-gnome_3.5.1-0precise2_i386.deb
gambas3-gb-eval-highlight_3.5.1-0precise2_all.deb
gambas3-gb-form_3.5.1-0precise2_all.deb
gambas3-gb-form-dialog_3.5.1-0precise2_all.deb
gambas3-gb-form-mdi_3.5.1-0precise2_all.deb
gambas3-gb-form-stock_3.5.1-0precise2_all.deb
gambas3-gb-gmp_3.5.1-0precise2_amd64.deb
gambas3-gb-gmp_3.5.1-0precise2_i386.deb
gambas3-gb-gsl_3.5.1-0precise2_amd64.deb
gambas3-gb-gsl_3.5.1-0precise2_i386.deb
gambas3-gb-gtk_3.5.1-0precise2_amd64.deb
gambas3-gb-gtk_3.5.1-0precise2_i386.deb
gambas3-gb-gtk-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-gtk-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-httpd_3.5.1-0precise2_amd64.deb
gambas3-gb-httpd_3.5.1-0precise2_i386.deb
gambas3-gb-image_3.5.1-0precise2_amd64.deb
gambas3-gb-image_3.5.1-0precise2_i386.deb
gambas3-gb-image-effect_3.5.1-0precise2_amd64.deb
gambas3-gb-image-effect_3.5.1-0precise2_i386.deb
gambas3-gb-image-imlib_3.5.1-0precise2_amd64.deb
gambas3-gb-image-imlib_3.5.1-0precise2_i386.deb
gambas3-gb-image-io_3.5.1-0precise2_amd64.deb
gambas3-gb-image-io_3.5.1-0precise2_i386.deb
gambas3-gb-jit_3.5.1-0precise2_amd64.deb
gambas3-gb-jit_3.5.1-0precise2_i386.deb
gambas3-gb-libxml_3.5.1-0precise2_amd64.deb
gambas3-gb-libxml_3.5.1-0precise2_i386.deb
gambas3-gb-logging_3.5.1-0precise2_all.deb
gambas3-gb-map_3.5.1-0precise2_all.deb
gambas3-gb-media_3.5.1-0precise2_amd64.deb
gambas3-gb-media_3.5.1-0precise2_i386.deb
gambas3-gb-memcached_3.5.1-0precise2_all.deb
gambas3-gb-mime_3.5.1-0precise2_amd64.deb
gambas3-gb-mime_3.5.1-0precise2_i386.deb
gambas3-gb-mysql_3.5.1-0precise2_amd64.deb
gambas3-gb-mysql_3.5.1-0precise2_i386.deb
gambas3-gb-ncurses_3.5.1-0precise2_amd64.deb
gambas3-gb-ncurses_3.5.1-0precise2_i386.deb
gambas3-gb-net_3.5.1-0precise2_amd64.deb
gambas3-gb-net_3.5.1-0precise2_i386.deb
gambas3-gb-net-curl_3.5.1-0precise2_amd64.deb
gambas3-gb-net-curl_3.5.1-0precise2_i386.deb
gambas3-gb-net-pop3_3.5.1-0precise2_amd64.deb
gambas3-gb-net-pop3_3.5.1-0precise2_i386.deb
gambas3-gb-net-smtp_3.5.1-0precise2_amd64.deb
gambas3-gb-net-smtp_3.5.1-0precise2_i386.deb
gambas3-gb-openal_3.5.1-0precise2_amd64.deb
gambas3-gb-openal_3.5.1-0precise2_i386.deb
gambas3-gb-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-glsl_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-glsl_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-glu_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-glu_3.5.1-0precise2_i386.deb
gambas3-gb-opengl-sge_3.5.1-0precise2_amd64.deb
gambas3-gb-opengl-sge_3.5.1-0precise2_i386.deb
gambas3-gb-openssl_3.5.1-0precise2_amd64.deb
gambas3-gb-openssl_3.5.1-0precise2_i386.deb
gambas3-gb-option_3.5.1-0precise2_amd64.deb
gambas3-gb-option_3.5.1-0precise2_i386.deb
gambas3-gb-pcre_3.5.1-0precise2_amd64.deb
gambas3-gb-pcre_3.5.1-0precise2_i386.deb
gambas3-gb-pdf_3.5.1-0precise2_amd64.deb
gambas3-gb-pdf_3.5.1-0precise2_i386.deb
gambas3-gb-qt4_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-ext_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-ext_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-opengl_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-opengl_3.5.1-0precise2_i386.deb
gambas3-gb-qt4-webkit_3.5.1-0precise2_amd64.deb
gambas3-gb-qt4-webkit_3.5.1-0precise2_i386.deb
gambas3-gb-report_3.5.1-0precise2_all.deb
gambas3-gb-sdl_3.5.1-0precise2_amd64.deb
gambas3-gb-sdl_3.5.1-0precise2_i386.deb
gambas3-gb-sdl-sound_3.5.1-0precise2_amd64.deb
gambas3-gb-sdl-sound_3.5.1-0precise2_i386.deb
gambas3-gb-settings_3.5.1-0precise2_all.deb
gambas3-gb-signal_3.5.1-0precise2_amd64.deb
gambas3-gb-signal_3.5.1-0precise2_i386.deb
gambas3-gb-v4l_3.5.1-0precise2_amd64.deb
gambas3-gb-v4l_3.5.1-0precise2_i386.deb
gambas3-gb-vb_3.5.1-0precise2_amd64.deb
gambas3-gb-vb_3.5.1-0precise2_i386.deb
gambas3-gb-web_3.5.1-0precise2_all.deb
gambas3-gb-xml_3.5.1-0precise2_amd64.deb
gambas3-gb-xml_3.5.1-0precise2_i386.deb
gambas3-gb-xml-html_3.5.1-0precise2_amd64.deb
gambas3-gb-xml-html_3.5.1-0precise2_i386.deb
gambas3-gb-xml-rpc_3.5.1-0precise2_all.deb
gambas3-gb-xml-xslt_3.5.1-0precise2_amd64.deb
gambas3-gb-xml-xslt_3.5.1-0precise2_i386.deb
gambas3-ide_3.5.1-0precise2_all.deb
gambas3-runtime_3.5.1-0precise2_amd64.deb
gambas3-runtime_3.5.1-0precise2_i386.deb
gambas3-scripter_3.5.1-0precise2_all.deb
When downloading add the following information to the output:
Thanks :D
Hi
I mirror different Debian versions, and would like to keep the old files for only the last distribution. I think that's the purpose of skip-clean directive, but can't get it to work.
I tested with differents line formats :
clean http://ftp.fr.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.fr.debian.org/debian/dists/wheezy/
OR
clean http://ftp.fr.debian.org/debian
clean http://security.debian.org/
skip-clean http://ftp.fr.debian.org/debian wheezy main contrib non-free
skip-clean http://security.debian.org/ wheezy/updates main contrib non-free
skip-clean http://ftp.fr.debian.org/debian/ wheezy-updates main contrib non-free
In the 2 cases, the whole "ftp.fr.debian.org/debian" is skipped.
Is there anything that I miss ? Or is it possible to do what I want with clean/skip-clean options ?
Cheers
NG
Hi,,
We've noticed that there is no InRelease file for Debian 8.2 in
ftp://ftp.se.debian.org/debian/dists/jessie/
but there was such file for Debian 8.1 in this dir and apt-mirror seems to skip cleaning InRelease files; skipping code in apt-mirror (*):
foreach ( keys %urls_to_download )
{
s[^(\w+)://][];
s[~][%7E]g if get_variable("tilde");
$skipclean{$} = 1;
$skipclean{$} = 1 if s[.bz2$][];
$skipclean{$_} = 1 if s[.xz$][];
}
The result of mirroring
ftp://ftp.se.debian.org/debian/dists/jessie/
with apt-mirror has now old InRelease (left from Debian 8.1) and new Release and Release.gpg (from Debian 8.2) which generates apt-get Hash Sum errors because apt-get seems to process InRelease if found, not Release+Release.gpg.
Also, the code (*) seems to have a bug: line with "$skipclean{$_} = 1;" already marks all the files from urls_to_download probably and lines
$skipclean{$_} = 1 if s[\.gz$][];
$skipclean{$_} = 1 if s[\.bz2$][];
$skipclean{$_} = 1 if s[\.xz$][];
do nothing (maybe the intention was to set it to 0 there to allow cleaning compressed stuff if compression format changes)?
What is the whole urls_to_download cleaning skipping for? What if repo maintainers change compression format from bz2 to xz or switch from InRelease to Release+Release.gpg only or from Release+Release.gpg to InRelease only?
Is it a problem to remove whole (*) code and let apt-mirror to sync all files from urls_to_download 1:1 without leaving old stuff there?
Regards,
Pawel
Fetched 437 kB in 1s (289 kB/s)
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-updates/main/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-updates/main/binary-i386/Packages Hash Sum mismatch
Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/source/Sources Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/source/Sources Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/source/Sources Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/main/binary-i386/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/non-free/binary-i386/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-backports/contrib/binary-i386/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/source/Sources Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/non-free/source/Sources Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages Hash Sum mismatch
W: Failed to fetch http://10.18.25.30/debian/dists/jessie-proposed-updates/main/binary-i386/Packages Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
And I delete the file will be ok!
rm -rf debian/dists/jessie-updates/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-updates/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/main/source/Sources.bz2
rm -rf debian/dists/jessie-backports/non-free/source/Sources.bz2
rm -rf debian/dists/jessie-backports/contrib/source/Sources.bz2
rm -rf debian/dists/jessie-backports/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/non-free/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/contrib/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-backports/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/non-free/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-backports/contrib/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/main/source/Sources.bz2
rm -rf debian/dists/jessie-proposed-updates/non-free/source/Sources.bz2
rm -rf debian/dists/jessie-proposed-updates/main/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages.bz2
rm -rf debian/dists/jessie-proposed-updates/main/binary-i386/Packages.bz2
rm -rf debian/dists/jessie-updates/main/binary-amd64/Packages
rm -rf debian/dists/jessie-updates/main/binary-i386/Packages
rm -rf debian/dists/jessie-backports/main/source/Sources
rm -rf debian/dists/jessie-backports/non-free/source/Sources
rm -rf debian/dists/jessie-backports/contrib/source/Sources
rm -rf debian/dists/jessie-backports/main/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/non-free/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/contrib/binary-amd64/Packages
rm -rf debian/dists/jessie-backports/main/binary-i386/Packages
rm -rf debian/dists/jessie-backports/non-free/binary-i386/Packages
rm -rf debian/dists/jessie-backports/contrib/binary-i386/Packages
rm -rf debian/dists/jessie-proposed-updates/main/source/Sources
rm -rf debian/dists/jessie-proposed-updates/non-free/source/Sources
rm -rf debian/dists/jessie-proposed-updates/main/binary-amd64/Packages
rm -rf debian/dists/jessie-proposed-updates/non-free/binary-amd64/Packages
rm -rf debian/dists/jessie-proposed-updates/main/binary-i386/Packages
i would love to see an exclude dirs in config for clean.sh
coz when i pull the installer-amd64 / i386 directorys via postmirror.sh script
i don´t want to redownload them each time clean.sh runs ,
or how about adding them to apt-mirror
great work btw.
thx
The following codesegment should be:
my $lang;
foreach $lang ( "ast", "bg", "bs", "ca", "cs", "da", "de", "de_DE", "el", "en", "en_AU", "en_CA", "en_GB", "eo", "es", "eu", "fa", "fi", "fr", "fr_FR", "gl", "hr", "hu", "id", "is", "it", "ja", "ka", "km", "ko", "lt", "ms", "my", "nb", "nl", "no_NB", "oc", "pl", "pt", "pt_BR", "ro", "ru", "si", "sk", "sl", "sq", "sr", "sv", "th", "tr", "uk", "vi", "zh_CN", "zh_HK", "zh_TW" )
{
add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang );
add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang . ".gz" );
add_url_to_download( $url . $_ . "/i18n/Translation-" . $lang . ".bz2" );
}
add_url_to_download( $url . $_ . "/i18n/Index" );
}
On ftp.at.debian.org only Packages.gz, Packages.xz as well as Sources.gz and Sources.xz are available for download [1,2].
Apt-mirror relies on the availability of Packages.bz2 and Sources.bz2. Since those do not exist on ftp.at.debian.org, apt-mirror seems to autocreate them. They appear on the filesystem with a timestamp matching the last apt-mirror run.
If a client issues an apt-get update
, those generated files yield a Hash Sum mismatch error.
If Packages.bz2 or Sources.bz2 is not available on the server side, it should not be generated by apt-mirror. Furthermore, once Packages.bz2 or Sources.bz2 disappear on the server side at one point in time, it should also disappear on the mirror side.
affected version: 0.4.9-1
[1] ftp://ftp.at.debian.org/debian/dists/jessie/main/binary-amd64/
[2] ftp://ftp.at.debian.org/debian/dists/jessie/main/source/
I'm doing some 'advanced' items in my postmirror.sh and I need access to the config variables from apt-mirror. Specifically in my case base_path, mirror_path, & var_path. Currently I added the following lines to apt-mirror for the variables to be in the environment for postmirror.sh:
$ENV{base_path}=get_variable( "base_path" );
$ENV{mirror_path}=get_variable( "mirror_path" );
$ENV{var_path}=get_variable( "var_path" );
But this may be better handled by something like the following code? (I'm not a perl programmer)
######################################################################################
## Setting Environment variables to be used post mirror
foreach my $key ( keys %config_variables )
{
$ENV{$key}=get_variable( "$key" );
}
Or is there a better way to get access to these variables from inside the postmirror.sh script?
Hi,
these files are listed in the Release
files but ignored by apt-mirror. I parse the Release
files and do a md5sum-check on the files listed there, leading to this error:
md5sum: main/dep11/Components-amd64.yml.xz: No such file or directory
Here is one example in the ubuntu repos: http://de.archive.ubuntu.com/ubuntu/dists/devel/main/dep11/Components-amd64.yml.xz
apt-mirror doesn't stop when it receives the university firewall's login page instead of the file it intends to download. This always happens when my quota for the day is over.
e.g., apt-repo/mirror/archive.ubuntu.com/ubuntu/dists/trusty/InRelease
<HTML><HEAD>
<meta http-equiv=pragma content=nocache>
<META HTTP-EQUIV=Expires CONTENT=-1>
<SCRIPT>
location.href="http://10.1.2.3:80/httpclient.html"
</SCRIPT>
</HEAD><BODY>
</BODY>
</HTML>
This invariably corrupts the repository, making apt-mirror to deem most of the repository's files unnecessary,
Downloading 425 index files using 4 threads...
Begin time: Thu Sep 1 01:00:01 2016
[4]... [3]... [2]... [1]... [0]...
End time: Thu Sep 1 01:00:23 2016
Processing tranlation indexes: [TTTTTTTTTTTTTTT]
Downloading 0 translation files using 0 threads...
Begin time: Thu Sep 1 01:00:23 2016
[0]...
End time: Thu Sep 1 01:00:23 2016
Processing indexes: [SSSSSPPPPPPPPPPPPPPP]
0 bytes will be downloaded into archive.
Downloading 0 archive files using 0 threads...
Begin time: Thu Sep 1 01:00:24 2016
[0]...
End time: Thu Sep 1 01:00:24 2016
310.3 GiB in 294483 files and 29491 directories can be freed.
Run apt-repo/var/clean.sh for this purpose.
Running the Post Mirror script ...
(apt-repo/var/postmirror.sh)
Post Mirror script has completed. See above output for any possible errors.
Could you please suggest solutions to stop apt-mirror in such a scenario ?
When manually started from the terminal, it seems apt-mirror is easy to stop using ctrl+c.
However, if apt-mirror is started via crontab it must be terminated via:
pkill -x -f '/bin/sh -c /usr/bin/apt-mirror > /opt/apt-mirror/mirror/archive.ubuntu.com/ubuntu/apt-mirror.log' --signal HUP
kill $( pidof wget )
or some other variant of these commands; since there is nothing to ctrl+c, there is no processes actually called apt-mirror, and because killing the apt-mirror process does not kill the 20 or so threads it created.
It would be nice if a feature were implement where one could just type:
apt-mirror stop
And halt the entire process.
Multiple warnings about uninitialized value $config{"options"}
I am finding that on my Ubuntu 12.04, certain dependency packages are listed as available in the update manager, but don't update because (it says) they can't be downloaded:
linux-generic
linux-headers-generic
linux-image-generic
The description of that first package, for example, is:
"This package will always depend on the latest complete generic Linux kernel available."
What's the story here? Is there something that can be done about this?
/DA
Contents of the associated directory for linux-generic:
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal-eol-upgrade_3.5.0.44.50_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring-eol-upgrade_3.8.0.34.34_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.23.25_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.23.25_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.23.25_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.57.68_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_amd64.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_i386.deb
mirror/us.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal_3.5.0.44.50_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-quantal/linux-generic-lts-quantal-eol-upgrade_3.5.0.44.50_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring_3.8.0.34.34_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-raring/linux-generic-lts-raring-eol-upgrade_3.8.0.34.34_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic_3.2.0.57.68_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-generic-pae_3.2.0.57.68_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_amd64.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy-eol-upgrade_3.11.0.14.13_i386.deb
mirror/security.ubuntu.com/ubuntu/pool/main/l/linux-meta-lts-saucy/linux-generic-lts-saucy_3.11.0.14.13_amd64.deb
when I run the command:
apt-mirror
one of the messages is:
Processing tranlation indexes: [T]
but it seems that the word 'tranlation' should be 'translation'
I'm using apt-mirror in a debian testing. version:
$ dpkg -l apt-mirror
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Nome Versão Arquitectura Descrição
+++-==============-============-============-==================================
ii apt-mirror 0.5.1-1 all APT sources mirroring tool
I'll like to know is it's possible to update an older (apt-)mirror on a local server using a recent one witout creating the whole new tree structure and copy a second repository right next to the old one.
Is that possible, is it DNS related?
The regex is incorrect:
Change: $uri =~ s&:\d+/&/&; # and port information
To:
A for instance is the Raspbian repo.
Processing indexes: [SPPPPPPPPPPPPPPPPsh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/main/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/contrib/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/non-free/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
sh: 1: cannot open mirrordirector.raspbian.org/raspbian///dists/jessie/rpi/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 769.
]
I see that it is expecting "Packages.gz" when this repo contains "Contents-armhf.gz". Don't know if this is something that can be added easily or not but it would be great to have this is a functionality.
Hello, I need to be able to define what the apt-mirror base directory is via an argument.
Currently, it's hard-coded to /var/spool/apt-mirror
but I need to use different directories instead to ease technical complications caused by the hard-coded path.
Thanks,
Wayne
OS: ubuntu16.04(apt-mirror default version 0.5.3.1)
apt-get remove apt-mirror && apt-get purge apt-mirror
I download latest version(0.5.4) from github and make install
this is my mirror.list
set base_path /var/spool/apt-mirror
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
set postmirror_script $var_path/postmirror.sh
set defaultarch amd64
set run_postmirror 0
set nthreads 20
set limit_rate 100m
set _tilde 0
# Use --unlink with wget (for use with hardlinked directories)
set unlink 1
set use_proxy off
set http_proxy 127.0.0.1:3128
set proxy_user user
set proxy_password password
deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main
# deb-src http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main
clean http://ubuntu-cloud.archive.canonical.com/ubuntu
then I run apt-mirror
output:
root@dev:~# apt-mirror
Downloading 14 index files using 14 threads...
Begin time: Mon Aug 28 11:13:31 2017
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Aug 28 11:13:31 2017
Processing translation indexes: [T]
Downloading 0 translation files using 0 threads...
Begin time: Mon Aug 28 11:13:31 2017
[0]...
End time: Mon Aug 28 11:13:31 2017
Processing DEP-11 indexes: [D]
Downloading 0 dep11 files using 0 threads...
Begin time: Mon Aug 28 11:13:31 2017
[0]...
End time: Mon Aug 28 11:13:31 2017
Processing indexes: [P]
242.4 MiB will be downloaded into archive.
Downloading 735 archive files using 20 threads...
Begin time: Mon Aug 28 11:13:31 2017
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Aug 28 11:13:32 2017
0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.
root@dev:~#
It seems OK, but when I uncomment the deb-src
line and run apt-mirror
Output:
root@dev:~# apt-mirror
Downloading 52 index files using 20 threads...
Begin time: Mon Aug 28 11:21:43 2017
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Aug 28 11:21:43 2017
Processing translation indexes: [TTTT]
Downloading 0 translation files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]...
End time: Mon Aug 28 11:21:43 2017
Processing DEP-11 indexes: [DDDD]
Downloading 0 dep11 files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]...
End time: Mon Aug 28 11:21:43 2017
apt-mirror: can't open index ubuntu-cloud.archive.canonical.com/ubuntu//dists/xenial-updates/ocata/main/source/Sources in process_index at /usr/local/bin/apt-mirror line 800, <STREAM> chunk 872.
Processing indexes: [SPPPP]
0 bytes will be downloaded into archive.
Downloading 0 archive files using 0 threads...
Begin time: Mon Aug 28 11:21:43 2017
[0]...
End time: Mon Aug 28 11:21:43 2017
0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.
root@dev:~#
I wander if there is any problem
The backend downloader for apt-mirror is wget. Does it make sense to add support for aria2 or axel as optional downloaders?
wget is a single-threaded downloader, though apt-mirror has a lot of code to spawn n-threads of wget. From what I can read in the code, it pulls down the indexes and then splits up the to-be-downloaded URIs into archive-urls.n which are fed to individual wget processes. Not bad, but it only helps where each connection to the server is throttled. It does nothing towards pulling the same file with multiple connections to one server, or to pull from multiple servers. The plus side of using wget is that it is installed by default on most Linux servers and is a well-known tool.
aria2 or axel, on the other hand, are multi-threaded downloaders to begin with. All the logic of multi-threading the download list can be handled by one of these instead of apt-mirror. Due to wget's ubiquity, you aren't going trash that code, but I imagine that adding support for aria2 or axel should be fairly straight-forward.
The main benefit is that these downloaders can pull the same file from multiple servers or the same file with multiple connections. apt-mirror with wget cannot do either. This would be a substantial benefit where we use apt-mirror with a really low-end connection.
Glad to see apt-mirror getting some polish!
apt-mirror currently downloads only the following Packages files in dists/<distribution>/<component>/binary-<arch>/
:
and the following Sources files in dists/<distribution>/<component>/source/
:
At least on ftp.at.debian.org, Packages.xz and Sources.xz are available too – apt-mirror should fetch those.
affected version: 0.4.9-1
If i have both deb-i386 http://
and deb-amd64 http://
(same site) in /etc/apt/mirror.list
, apt-mirror always raises error like below. But it works fine if i have only one arch (e.g. deb http://...
).
apt-mirror: can't open index <site>/debian//dists/jessie/jessie/binary-i386/Packages in process_index at /usr/bin/apt-mirror line 800, <STREAM> chunk 576.
Processing indexes: [PPapt-mirror: can't open index <site>/debian//dists/stretch/stretch/binary-i386/Packages in process_index at /usr/bin/apt-mirror line 800.
PPPPPP]
My /etc/apt/mirror.list
:
############# config ##################
set base_path /opt/apt-mirror
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
# set postmirror_script $var_path/postmirror.sh
set run_postmirror 0
set nthreads 10
set _tilde 0
############# end config ##############
# Debian 8
deb-i386 https://<site>/debian jessie jessie
deb https://<site>/debian jessie jessie
# Debian 9
deb-i386 https://<site>/debian stretch stretch
deb https://<site>/debian stretch stretch
# Ubuntu 14.04 LTS
deb-i386 https://<site>/ubuntu trusty trusty
deb https://<site>/ubuntu trusty trusty
# Ubuntu 16.04 LTS
deb-i386 https://<site>/ubuntu xenial xenial
deb https://<site>/ubuntu xenial xenial
Firstly I know that apt checks the signature before installing, that isn't the point of this issue - read on.
Apt-mirror tool doesn't seem to check the checksum of the downloaded file vs the checksum of the file as is reported in the Release file. Nor does it check the checksum of the previously downloaded file on disk vs the file on the remote server to ensure the local version isn't corrupt.
I believe this is necessary in order to ensure that you have mirrored a valid repository and that failures will not occour later when you attempt to install the packages. Also in case of local filesystem corruption you are able to repair the repository without needing to download everything again.
It is evident that this was intended at one point but does not appear to be fully implemented.
When I try to create a mirror of cloudera-cdh5, I found this error.
The apt list file is under /etc/apt/sources.list.d/cloudera-cdh5.list
on my system, and the content is :
deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
deb-src http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
I try to do the mirror by following command and got errors:
vagrant@ubuntu:~$ sudo apt-mirror /etc/apt/sources.list.d/cloudera-cdh5.list
Downloading 15 index files using 15 threads...
Begin time: Mon Apr 14 17:04:35 2014
[15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Apr 14 17:04:36 2014
Proceed indexes: [SPsh: 1: cannot open [arch=amd64]//dists/http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/precise-cdh5/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in proceed_index_gz at /usr/bin/apt-mirror line 460.
vagrant@ubuntu:~$
If I have to change the apt .list to the form below to make it work:
deb-amd64 http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
deb-src http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh precise-cdh5 contrib
Could you add support of [arch=amd64]
-like format? Thanks.
The mirror.list configuration supports many options, the file is well commented explaining each option.
mirror.list contains a single explanatory comment. The manpage is, at best, severly overstating how "well commented" mirror.list is.
I just build new mirror for Ubuntu 16.04 Xenial and apt-mirror won't download the by-hash directory at all.
For example :
http://re.archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/by-hash/
My local mirror after update it shows "404 not found"
Thx for your support
apt-mirror will attempt to download both gzip and bzip2 compressed indexes but is hard coded to only attempt to parse indexes compressed with gzip. As a result, apt-mirror will fail to mirror a repository that does not publish gzip compressed indexes.
apt-mirror should attempt to download gzip, bzip2, lzma, and uncompressed indexes and be able to successfully mirror a repository if any one format is provided.
When I try to mirror this specific architecture debian repo, it fails because apt-mirror recognise deb-armhf as deb-arm.
Why armhf is not available on arch specific support?
With apt-mirror
from mpesari (which fixes #56) and an Ubuntu Xenial repo:
Processing indexes: [PPPPPPPPPPPPPsh: 1: cannot open archive.canonical.com/ubuntu//dists/xenial/partner/debian-installer/binary-amd64/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 773.
Psh: 1: cannot open archive.canonical.com/ubuntu//dists/xenial/partner/debian-installer/binary-i386/Packages.gz: No such file
apt-mirror: can't open index in process_index_gz at /usr/bin/apt-mirror line 773.
]
In /etc/apt/mirror.list
:
deb http://archive.canonical.com/ubuntu xenial partner
deb-i386 http://archive.canonical.com/ubuntu xenial partner
clean http://archive.canonical.com
I can not find any doc or examples to tell how to configurate the defaultarch in mirror.list
I want to mirror both i386 and amd64
I just do the follow steps
set defaultarch i386
and run sudo apt-mirror
set defaultarch amd64
and run sudo apt-mirror
:(
We use apt-mirror for a local mirror. We want to keep some history for our mirrors, using hardlinks between two "snapshots".
The default behavior of wget is to clobber existing files if they changed. This is not something good for our setup since we want to be sure that NO file will change in previous snapshots.
We would like to use the wget option --unlink
in apt-mirror. I'm working on a patch to enable a new variable for this.
Hi @bholtsclaw
Thank you for your work behind apt-mirror maintenance, could you please provide a prioritized list of pending bugs/feat from people can start contributing?
Thank you.
I can't figure out how to specify a path with spaces in it (in mirror.list).
Enclosing in double quotes -> the double quotes are literally a part of the path.
Backslash before space -> the backslash becomes a part of the path.
Add documentation that describes the configuration options and their default values.
( Mainly what is there does not cover the configuration options and what they do, only shows the default and one limited example. )
I'm using apt-mirror on Ubuntu 16.04, and some of the repositories I'm mirroring does not contain MD5sums in the Packages files. This results in warnings like this:
Global symbol "$package" requires explicit package name (did you forget to declare "my $package"?) at /usr/bin/apt-mirror line 787
Line 787 in Ubuntu 16.04 corresponds to line 798 in version 0.5.2.
print FILES_MD5 $lines{"MD5sum:"} . " " . remove_double_slashes( $path . "/" . $lines{"Filename:"} ) . "\n";
Appending if defined $lines{"MD5sum:"}
to the line silences the error. However, you probably shouldn't silence all these possibly missing checksums. Maybe you could add some logic to warn if there isn't at least one checksum in the file?
Would it be possible to support repos that use SSL keys to authenticate
nginx repo comes to mind.
Hello,
If I add too many repos, more than 50, then apt-mirror crashes when downloading:
Processing indexes: [SSSSSSSSSSPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
6.5 GiB will be downloaded into archive.
Downloading 931 archive files using 20 threads...
Use of uninitialized value $pid in numeric lt (<) at /usr/bin/apt-mirror line 252.
Use of uninitialized value $pid in numeric eq (==) at /usr/bin/apt-mirror line 254.
Thank you in advance for your help.
I just build new mirror for Ubuntu 16.04 Xenial and apt-mirror won't download the dep11 directory at all.
The source has:
http://ca.archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/
My local mirror after update it shows "404 not found"
Of all the forks I've found, @kimmel has the most extensive changes, primarily in the form of refactoring, cleanup, and conformance to perlcritic:
https://github.com/kimmel/apt-mirror-fork
I've taken a quick look, but not long enough to know if the changes should be merged here.
i note this code in copy_file()
if ( get_variable("unlink") == 1 )
{
if ( compare( $from, $to ) != 0 ) { unlink($to); }
}
unless ( copy( $from, $to ) )
{
warn("apt-mirror: can't copy $from to $to");
return;
}
checks to see if the files are the same before unlinking, but then it goes ahead and tries to rewrite the file in any case. i can't figure out what sense that makes: if you are always going to overwrite the file, then why bother doing the comparison before unlinking, if unlink=1? why not just stat the $to file and unlink it if it exists instead?
(interesting, but totally beside the point: if the $to file doesn't exist, compare() should return -1 and so the code willl try unlinking the nonexistent file, if i read the perldoc right. for that matter, you could probably dispense with even trying to stat() and just unlink() away, since there's no check for an error, so "no such file" shouldn't hurt)
not trying to criticize, since i can't even begin to approach the awesomeness that went into creating this script. only reason i am bringing this up is because i get a permissions failure here when copy() tries to overwrite the file that didn't get unlink()-ed because compare() == 0. better to skip that check, for me. i actually just rewrote that bit to always do the comparison and to skip the overwriting, if the files compare equal, whether any unlinking is to be done or not (although this left me with one more place where the script still failed on me: utime() failed on a file i didn't have ownership of). anyways, i've attached my patch, if anybody has use for it. (change the relevant die, from the original script, to warn if you also don't care about utime() failing, though i'm not including that in the attached, since that maybe makes the script unreliable for mirroring archives where timestamp fidelity is critical, like for rsync-ing or what have you. seems to me that apt just cares about checksums, but i may end up finding out the hard way if not. also, anybody who knows Perl, please fix the bugs and stylistic infelicities and share, so i can update my copy, thanks. [note that i left some whitespace glitches in place to minimize the patch. feel free to re-indent])
Hello,
apt-mirror doesn't mirror "InRelease" files from repositories.
These files are mandatory for a network installation of Debian Wheezy.
Regards,
[Problem partly solved, see next post. The failing download is not related to the error message.]
I am repeatedly updating the mirror, and getting exactly the same result every time (see below). It seems that there are some files to be downloaded, but they must be failing (?) which I'm guessing is related to the "gzip: stdin: not in gzip format" error. I cut the threads down from 20 to 5 to see if it would help, but the results do not change. (Same number of "PPP"s before/between/after the errors.)
I see that "apt-get update" seems to sometimes produce this error:
https://answers.launchpad.net/ubuntu/+question/20750
Also, this same error reported on the web in cases of a corrupted tar archive:
http://fogproject.org/forum/threads/gzip-stdin-not-in-gzip-format-error-on-ubuntu-server-12-04.791/
Here is one other apt-mirror report:
http://askubuntu.com/questions/283070/apt-mirror-error-qzip-stdin-not-in-gzip-format
Any idea what's happening/ how I can fix it (or even identify the problem files/repository)?
/DA
sudo -u apt-mirror apt-mirror
Downloading 502 index files using 5 threads...
Begin time: Thu Dec 19 23:32:43 2013
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Thu Dec 19 23:33:05 2013
Proceed indexes: [PPPPPPP
gzip: stdin: not in gzip format
gzip: stdin: not in gzip format
PPPPPPP
gzip: stdin: not in gzip format
gzip: stdin: not in gzip format
PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
292.6 MiB will be downloaded into archive.
Downloading 16 archive files using 5 threads...
Begin time: Thu Dec 19 23:33:17 2013
[5]... [4]... [3]... [2]... [1]... [0]...
End time: Thu Dec 19 23:33:18 2013
0.0 bytes in 0 files and 0 directories can be freed.
Run /media/MAXTOR-300G/apt-mirror/var/clean.sh for this purpose.
Running the Post Mirror script ...
(/media/MAXTOR-300G/apt-mirror/var/postmirror.sh)
Removing 0 unnecessary files [0 bytes]...
done.
Removing 0 unnecessary directories...
done.
Post Mirror script has completed. See above output for any possible errors.
exec 'wget', '--no-cache', '--limit-rate=' . get_variable("limit_rate"), '-t', '5', '-r', '-N', '-c', '-l', 'inf', '-o', get_variable("var_path") . "/$stage-log.$i", '-i', get_variable("var_path") . "/$stage-urls.$i",
ADD OPTION '-c'
At a stop, there is no continuation or broken download resume.
The file remains not finished!
There is a value called _tilde that is set to 0 in the config file (mirror.list). There is however no mentioning of it in the man page nor in the code commentary. It would be useful to know what the value does/change. A short mention in the man page or the README.md would not go amiss.
I'm using apt-mirror for Debian repositories mirroring and even contributed a packaged version for Solaris...
In certain circumstances, such as when using live-build, the file Contents is required in some nodes of the repository tree, e.g. debian/dists/testing/main/.
Even though there is some processing related to this kind of file, it's not complete. That is the reason for which I propose a patch to completely acquire the required Contents file and, for completeness sake, I added also the acquisition of the InRelease file.
Thank you in advance for considering this patch
Note that I already contacted Brandon on this issue and he suggested to create a pull. However, I didn't succeeded to create a branch or I don't really understand how to create a pull. This is why I put the patch here:
*** /usr/bin/apt-mirror 2013-03-03 06:06:55.000000000 +0100
--- /home/peter/org/usr/bin/apt-mirror 2014-01-27 13:54:29.000000000 +0100
***************
*** 362,371 ****
$url = $uri . "/dists/" . $distribution . "/";
add_url_to_download($url . "Release");
add_url_to_download($url . "Release.gpg");
! if (get_variable("_contents")) {
! add_url_to_download($url . "Contents-" . $arch . ".gz");
! add_url_to_download($url . "Contents-" . $arch . ".bz2");
! }
foreach (@components) {
add_url_to_download($url . $_ . "/binary-" . $arch . "/Release");
--- 362,376 ----
$url = $uri . "/dists/" . $distribution . "/";
+ add_url_to_download($url . "InRelease");
add_url_to_download($url . "Release");
add_url_to_download($url . "Release.gpg");
!
! if (get_variable("_contents")) {
! foreach(@components)
! {
! add_url_to_download($url . $_ . "/Contents-" . $arch . ".gz");
! add_url_to_download($url . $_ . "/Contents-" . $arch . ".bz2");
! }
! }
foreach (@components) {
add_url_to_download($url . $_ . "/binary-" . $arch . "/Release");
Add option to config file to configure proxies.
This proxy can be parsed and added to low-level wget commands as an option.
Add option to check md5 and sha signatures of files, like debmirror does.
Hi,
we'd like to have support for lightweight mirroring to S3. As far as I can see now, this would entail pulling a list of packages from an S3 bucket (the target) and then pulling a list of packages from Upstream, merging both lists and dumping everything to S3 again.
Would this change be in scope for your project?
Cheers, Mark
If i'm not mistaken, apt-mirror
works by splitting the files to be downloaded up into batches of files / nthreads
files each. So if you've got 1000 files to download, and you've configured 2 threads, thread 0
gets 500 files and thread 1
gets 500 files.
The problem with this arrangement is that each set of files may be vastly different in size. For example, thread 0
might get 500 small command-line utilities and thread 1
might get 500 giant kernel images. So what ends up happening here is that all of your threads except one complete, and then you're left sitting there waiting for the last one to slowly pull down its massive pay-load all by itself. It gets especially bad if you have a per-thread download limit set — you could be waiting days for that one lonely instance of wget
to finish.
The package lists contain the size of each of the packages, right? Would it be possible to change the apportioning of files so that each wget
instance gets a roughly similar amount of data (rather than number of files) to fetch?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.