pirate/wikipedia-mirror

🌐 Guide and tools to run a full offline mirror of Wikipedia.org with three different approaches: Nginx caching proxy, Kiwix + ZIM dump, and MediaWiki/XOWA + XML dump

wikipedia-mirror's Issues

README.md wikipedia_en_all_novid_2018-10.zim (79GB, all English articles, no pictures/videos)

wikipedia_en_all_novid_2018-10.zim (79GB, all English articles, no pictures/videos)

should read
wikipedia_en_all_novid_2018-10.zim (79GB, all English articles and pictures, no videos)
or alternatively
wikipedia_en_all_nopic_2018-09.zim (37GB, all English articles, no pictures/videos)

Few remarks about the README

The README.md is a good explanation work :)

Here my few comments:

• We have permalinks for releases which could advantageously replace https://mirrors.dotsrc.org/kiwix/release/kiwix-tools/kiwix-tools_linux-x86_64-3.0.1.tar.gz by https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-x86_64.tar.gz
• I would recommend AFAP to recommend HTTPS over HTTP, so better https://download.kiwix.org/zim/wikipedia_en_all_novid.zim over https://download.kiwix.org/zim/wikipedia_en_all_novid.zim
• novid links will be soon be replaced by maxi, it is just a naming convention, so https://download.kiwix.org/zim/wikipedia_en_all_maxi.zim
• ZIM/Kiwix has fulltext to the contrary to what you have written.
• Wikipedia categories will come soon, this is still in alpha.
• About the list of ZIMs, Wikipedia list is a mirror of https://download.kiwix.org/zim/wikipedia/
• All the links on download.kiwix.org have a BitTorrent file, just add .torrent to any URL
• The publication is not regularly like you have noticed, but our goal is one time a month for most of them. Still working on it (We are searching for HW to support us).
• I would recommend to use permalinks everywhere, to avoid having to update the READMEmd

Can't load .env variables into .conf

This line generates the .conf from the template, but doesn't seem to load any of the sourced .env variables -- nginx -t reports them as invalid...

# Fill your config options into nginx.conf.template to create nginx.conf
envsubst \
    "$(printf '${%s} ' $(bash -c "compgen -A variable"))"\
    < "$CONFIG_DIR/nginx.conf.template" \
    > "$CONFIG_DIR/nginx.conf"

Is there a way to keep the local dump up to date?

Thank you for the guide. It would be helpful to include a way to keep the setup up to date periodically. I wish there was a way to to just download the 'diff' every month instead of complete dump.