kiwix/kiwix-build

Kiwix & openZIM build engine

Kiwix

Kiwix is an offline reader for Web content. It's especially thought to make Wikipedia available offline. This is done by reading the content of the project stored in a file format ZIM, a high compressed open format with additional meta-data.

COMPILATION INSTRUCTIONS

Take care, the paragraphs are about the target platforms. If you want to build Kiwix for Android on a GNU/Linux system, you should follow the instructions of the "Android" paragraph.

GNU/Linux

Install pre-requisties in your distro, eg, in Debian based:

sudo apt-get install zip pkg-config aptitude
sudo apt-get install libmagic-dev
sudo aptitude install libtool automake

Static (Probably what you need to do if you don't know)

Kiwix uses shared libraries only. A static build of Kiwix is a build which packages the dependencies. Command line tools (indexer, server, etc) are compiled statically.

Run automake

./autogen.sh

Run autoconf

./configure --enable-compileall --enable-staticbins --disable-android;

Download all dependencies

cd src/dependencies;
make;
cd ../..;

Reconfigure with dependencies (Gecko version)

./configure --enable-compileall --enable-staticbins --disable-android;

Compile

make;

Creates tarbal suitable for distribution (no deps)

make diststatic

• If you want to be able to run directly from your dev repository copy the "xulrunner" directory in the distributation tarball to the "kiwix" sub-directory of your developement tree.

• Don't try to install with "make install", if you want to have kiwix install on your system just copy the directory in a /usr/local/bin directory.

Dynamic

GNU/Linux build uses shared libraries by default. You will need the following dependencies (also -dev packages) to compile Kiwix for Linux (this list can vary a little bit depending of the GNU/Linux distribution):

• g++
• gcc
• autoconf
• automake
• libtool
• pkg-config
• liblzma
• libicu
• libmicrohttpd
• zlib
• libcrypto++
• cmake
• wget
• aria2
• libuuid
• libssl
• libzim
• libpugixml
• libctpp2
• xulrunner

Debian is the only distribution providing natively all these packages. Ubuntu provides most of them, excluding xulrunner and zimlib. You will have to download and install them separatly or run configure --with-static-dep=SELF. Then run the following commands:

Run automake

./autogen.sh;

Configure the compilation check --help for options. Most dependencies accept --with-dep=XX and with-static-dep=XX.a. Static version of libraries are used for building static binaries (server, indexer, etc) Use --with-dep=SELF or --with-static-dep=SELF to trigger fetch and build for the dependency.

./configure;

Compile

make;

Optionaly install on the filesystem

sudo make install;

Mac OSX Universal

WARNING: To build the Mac OS version you will have to install proprietary software which are free of charge. You will also need to build on an Apple Mac computer.

• Install OSX 10.6.5+ (proprietary)
• Install XCode 3.2.5+ (proprietary) - http://developer.apple.com/technologies/xcode.html (requires registration) http://download.kiwix.org/dev/xcode_3.2.5_and_ios_sdk_4.2_final.dmg
• Install Java
• Install MacPorts - http://www.macports.org/install.php

Configure Macports

Install the following tools and libraries

sudo port install autogen +universal pkgconfig +universal wget +universal gmake +universal coreutils +universal libidl +universal autoconf213 +universal icu +universal;

NOTE: the following commands seem to work better on OS X Yosemite, apparently 'universal' is less relevant and various packages are no longer available as universal options.

sudo port  install autogen pkgconfig  wget  gmake  coreutils  libidl  autoconf213  icu;
sudo port -v install aclocal automake libtool autoconf cmake imagemagick
./autogen.sh alt && ./configure --enable-compileall;
cd src/dependencies && make;
make clean;
./autogen.sh alt && ./configure --enable-compileall --enable-compileall --disable-dependency-tracking --with-target-arch=i386;
make;
make distmac;
make clean;
./autogen.sh alt && ./configure --enable-compileall --enable-compileall --disable-dependency-tracking --with-target-arch=x86_64;
make;
make distmac;
make universal;

Debugging tips:

• components file type can be either Mach-O dynamically linked shared library or Mach-O bundle but above instructions will create dynamic libraries.
• Shared Object (.so files on Linux) have .dylib extension on OSX.
• use dtruss to inspect program execution like strace.
• dyldinfo -lazy_bind | dyldinfo -bind components/zimAccessor.dylib |grep zim
• nm -gm components/zimAccessor.dylib | nm -u | nm -g
• otool -L libzim.dylib
• install_name_tool -change @executable_path/../libicuuc.dylib libicuuc.dylib kiwix-serve
• lldb

Android

Look at android/README

Windows

• Install Windows XP SP2+

• Install Visual Studio Express 2010

• Install 7-zip

• Install MozillaBuild 1.6

• Install ActivePerl

• Install Ruby

• Install NSIS 2.46

• Install nsis_locate

• Install nsis_uac

• Replace installed UAC.dll by new one.

• Install all software in default locations.

• Change your Windows PATH environment variable:

  • 7zip
  • NSIS
  • ruby
  • Perl
  • mozilla-build\msys\bin\

• Get shell from c:\mozilla-build\start-msvc10.bat

mkdir -p /c/slave/windows-32b
git clone git://git.code.sf.net/p/kiwix/kiwix kiwix
cd kiwix
./autogen.sh alt
./configure --disable-indexer --enable-jar
make win
make windist
make wininstaller

Contact

Email: [email protected] or [email protected]

Jabber: [email protected]

IRC: #kiwix on irc.freenode.net

You can use IRC web interface on http://chat.kiwix.org/

More... http://www.kiwix.org/wiki/Communication

LEGAL & DISCLAIMER

Read 'COPYING' file

kiwix-build's Issues

macOS: error encountered while building uuid

Here is the log

uuid_error.txt

increase xapian version dependency

Currently, kiwix-build builds xapian 1.4.0
When compiling zimwriterfs (required for mwoffliner to work), the compilation fails because it requires at least xapian 1.4.2

Expected: compile xapian 1.4.2 in kiwix-build, so that it allows to compile zimwriterfs too

Automatically install ninja

It's not compulsory, but would simplify the installation by avoiding some manual steps

Meson encountered an error in file meson.build, line 39, column 4:

Error encountered: For custom ctpp2_prefix_install you need a meson version >=0.31.0

root@deb9-demo:/opt/tools/kiwix-build# meson -v
0.37.1

Verify that Windows cross-compile works fine on Ubuntu 16.04

Desktop app crashes

Reno (at hackathon) was showing me that the Kiwix desktop app crashes a lot on OSX 10.9 (I think). It's on ZIM file open sometimes, and when it happens he can't get it to work again until he clears cache.

Alert for "downloading list of catalogs" is not dismissable

The pop up indicates "do not ask me again," which only registers with the app if you say "yes" and not if you say "no." So you have to dismiss it all the time. (Saw this on Reno's computer at the hackathon)

Cross-compile kiwix-tools on MIPS

This is to get kiwix-serve working on most OpenWRT devices

"TypeError: urlopen() got an unexpected keyword argument 'context'" when building on Debian Jessie

Workaround : edit kiwix-build.py to remove the "context" argument when urlopen() is called (line 418).

It seems to come from the version of python3 distributed with debian jessie (3.4.2-2)

how can I do a custom android app build on a file already downloaded

Publish kiwix-tools for i586 and i686 archictectures

From @holta on November 22, 2017 18:20

Currently on OLPC's 32-bit XO laptops in the developing world we are using:
http://download.kiwix.org/bin/0.9/kiwix-0.9-linux-i686.tar.bz2 (30-Oct-2014)

As this is more than 3 years old and failing with new content, we're also experimenting with:
http://download.kiwix.org/bin/0.10/kiwix-0.10-linux-i686.tar.bz2 (26-Oct-2016)

With essentially all new ZIM content coming online at the end of 2017 having an internal index in the ZIM file, the above strategies are no longer working :-(

Can compiler flags hopefully be set to publish kiwix-tools (kiwix-serve, kiwix-manage) for the i686 architecture as well? Ideally publishing to http://download.kiwix.org/nightly/ and also upholding the i686 tradition @ http://download.kiwix.org/bin/ ?

Context: 3 million of OLPC's XO laptops were manufactured, and several 100,000 are today in active use, where Wikipedia on Kiwix is often the most adored of all apps/content when it is installed! A special thank you to @georgejhunt & @darkenvy whose field community/integration has made this work a centerpiece in Haiti, Ghana, Mexico etc, where we distribute different versions of WikipediaFR, WikipediaHT, WikipediaEN and WikipediaES.

Copied from original issue: kiwix/kiwix-tools#112

macOS: error when build kiwix-tools natively

kiwix-build stopped when building kiwix-tools, seems to sdo with -lrt. I am building using apple-platforms branch.

The log is here:
log.txt

Fix dependency nightmare with icu4c and mingw32.

• With the recent update of glibc 2.26 (2017-08-02), the xlocale.h header has been removed (https://sourceware.org/glibc/wiki/Release/2.26#Removal_of_.27xlocale.h.27)
• icu4c 58.1 is using xlocale.h
• icu project has fix this in the release 60.1. => So we will have to use last icu version to compile on system with recent glibc. (Fedora 27 at least)
• icu4c 60.1 is using the function ResolveLocaleName on Windows
• mingw32 doesn't export this function in kernel32. There is patch for this (https://sourceforge.net/p/mingw-w64/mingw-w64/ci/6dc5579d5e71f2662316416502b01c2d3ace0232) but no release has been made yet.
• Once mingw32 made a release, distributions will have to update their packages and travis (who uses ubuntu) will have to update its image to allow use to compile all libzim/kiwix stack again for win32.

Would it be a good idea to split kiwix-build.py into multiple files?

Currently kiwix-build.py is too huge, I find it hard to navigate. And I cannot put work on different classes in different commits. (i.e. if I added something to both iOS_sdk and iOSTargetInfo, I have to commit them together)

I think it would be a better idea to separate them into files like TargetInfo.py, Toolchain.py, etc.

Impossible to build custom apps

It seems there is a bug with Travis and the Requests. I have open a ticket here:
travis-ci/travis-ci#7982

See also #58 for the improvements.

build custom apk - KeyError: 'VERSION_CODE'

Continue from #79 with pip3 install pillow

./kiwix-build.py --verbose --target-platform android_arm --android-custom-app wikimedar kiwix-android-custom --zim-file-url https://download.kiwix.org/zim/wikipedia/wikipedia_ar_medicine_2017-08.zim

download_zim kiwix-android-custom : Sha256 for wikipedia_ar_medicine_2017-08.zim not set, do no verify download
OK
compile kiwix-android-custom : ERROR
Traceback (most recent call last):
File "./kiwix-build.py", line 990, in
builder.run()
File "./kiwix-build.py", line 928, in run
self.build()
File "./kiwix-build.py", line 918, in build
builder.build()
File "/opt/cesar-build/kiwix-build/dependencies.py", line 447, in build
self.command('compile', self._compile)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 206, in command
return self.target.command(*args, **kwargs)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 50, in command
ret = function(*args, context=context)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 369, in _compile
gradle_option=self.gradle_option)
File "/opt/cesar-build/kiwix-build/dependencies.py", line 423, in gradle_option
version_code=os.environ['VERSION_CODE'],
File "/opt/cesar-build/lib/python3.5/os.py", line 725, in getitem
raise KeyError(key) from None
KeyError: 'VERSION_CODE'

When is the optimal time to run travis cron jobs?

Travis has a concurrent job limit of I think 5 tasks per org. Kiwix build can delay pull request buids by around an hour. Might it be a better idea to schedule these builds at less active times?

Android nightlies are broken

They are not published anymore

macOS: ld: library not found for -lrt

on macOS, build natively, encountered the following error
lrt_error.txt

Quote comment by @mgautierfr in issue #9

For what I see, clang cannot found librt. The flag -lr comes from the xapian pkg-config if the configure detect (and found) the timer_create function. This part was a bit buggy and I fix it with the e6673e2.
Try to remove all xapian and kiwix-tools stuff (SOURCE and BUILD) and launch a new compilation.
If it doesn't work, you will have to fix xapian (first find if you really need librt, and then find either why xapian add it or why kiwix-tools cannot found it)

The question is:

• I assume @mgautierfr means -lrt is carried from xapian. (There was no flag -lr) If this is true, how come xapian build successfully?
• How do I know if librt is needed? I don't know what it does and how xapian use it

No module named 'PIL' when building apk

on ubuntu 16.4 VM

install dependencies with following script

#!/usr/bin/env bash
# Python3, pip3, ninja
apt-get install python3-pip virtualenv && virtualenv -p python3 ./ && source ./bin/activate && pip3 install meson && hash -r && git clone git://github.com/ninja-build/ninja.git && cd ninja && git checkout release && ./configure.py --bootstrap && mkdir -p ../bin && cp ninja ../bin && cd .. && source ./bin/activate

clone

https://github.com/nataren/kiwix-build
https://github.com/nataren/kiwix-android-custom

(differ only in that embed_zim is true)

run

python ./kiwix-build.py --verbose --target-platform android_arm --android-custom-app wikimedar kiwix-android-custom --zim-file-url https://download.kiwix.org/zim/wikipedia/wikipedia_ar_medicine_2017-08.zim

result:

Installing /opt/cesar-build/kiwix-build/BUILD_android_arm/kiwix-lib/kiwix.pc to /opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib/x86_64-linux-gnu/pkgconfig OK build kiwix-android : SKIP build kiwix-android-custom : configure kiwix-android-custom : run command './gen-custom-android-directory.py wikimedar --output-dir /opt/cesar-build/kiwix-build/BUILD_android_arm/kiwix-android-custom_wikimedar/custom/wikimedar' current directory is '/opt/cesar-build/kiwix-build/SOURCE/kiwix-android-custom' env is : CXX : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin/arm-linux-androideabi-g++' PS1 : '(cesar-build) \\[\\e]0;\\u@\\h: \\w\\a\\]${debian_chroot:+($debian_chroot)}\\u@\\h:\\w\\$ ' CXXFLAGS : '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' PATH : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin:/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/bin:/usr/lib/ccache:/opt/cesar-build/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' LESSCLOSE : '/usr/bin/lesspipe %s %s' PWD : '/opt/cesar-build/kiwix-build' SSH_TTY : '/dev/pts/0' HOME : '/root' PKG_CONFIG_PATH : ':/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib/x86_64-linux-gnu/pkgconfig' SSH_CONNECTION : '192.168.0.56 60203 10.100.0.9 22' MAIL : '/var/mail/root' SHELL : '/bin/bash' SSH_CLIENT : '192.168.0.56 60203 22' LD_LIBRARY_PATH : ':/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib:/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib64' SHLVL : '1' TERM : 'xterm' OLDPWD : '/opt/cesar-build' LOGNAME : 'root' CC : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin/arm-linux-androideabi-gcc' XDG_SESSION_ID : '49' CFLAGS : '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' USER : 'root' LESSOPEN : '| /usr/bin/lesspipe %s' LDFLAGS : '-L/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib -L/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' VIRTUAL_ENV : '/opt/cesar-build' NDK_DEBUG : '0' XDG_RUNTIME_DIR : '/run/user/0' LS_COLORS : 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' LANG : 'en_US.UTF-8' _ : '/opt/cesar-build/bin/python' ANDROID_HOME : '/opt/cesar-build/kiwix-build/TOOLCHAINS/android-sdk-r25.2.3' PKG_CONFIG_LIBDIR : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot/lib/pkgconfig' CPPFLAGS : '-I/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/include ' Traceback (most recent call last): File "./gen-custom-android-directory.py", line 28, in <module> from PIL import Image ImportError: No module named 'PIL' ERROR Stopping build due to errors

Bring back the Android universal/multiarch APK

This is essential for people not relying on the Google Play store. This should work for both custom apps and standard Android app.

Integrate fat lib building procedure for kiwix-lib targeting iOS

A lot of times we need to use lipo to combine libs for multiple arches into one fat lib. It would be a great idea to add this function to kiwix-build.

requirements_build_custom_app.txt is missing in git repo

Impossible to go further with that documentation:
https://github.com/kiwix/kiwix-build/wiki/Build-and-publish-a-custom-app

Limit concurrent kiwix-build jobs to prevent 40 min backlog

We should limit the concurrent kiwix-build jobs in travis to 3-4 so that other repos pull requests are not blocked for 50 mins+ as was the case for me tonight.

Make official/stable releases with travis

based on git tags

Crash by trying to build custom app

$ ./kiwix-build.py --target-platform android_arm --android-custom-app wikimed --zim-file-size $ZIM_SIZE kiwix-android-custom
[INSTALL PACKAGES]
 - automake : SKIP
 - cmake : SKIP
 - git : SKIP
 - subversion : SKIP
 - ccache : SKIP
 - pkg-config : SKIP
 - gcc : SKIP
 - default-jdk : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources for toolchain android-ndk :
  download android-ndk : SKIP
  extract android-ndk : SKIP
prepare sources for toolchain android-sdk :
  download android-sdk : SKIP
  extract android-sdk : SKIP
prepare sources Gradle :
  download Gradle : SKIP
  extract Gradle : SKIP
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : SKIP
  patch pugixml : SKIP
prepare sources zlib :
  download zlib : SKIP
  extract zlib : SKIP
  patch zlib : SKIP
prepare sources lzma :
  download lzma : SKIP
  extract lzma : SKIP
prepare sources uuid :
  download uuid : SKIP
  extract uuid : SKIP
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : SKIP
  patch xapian-core : SKIP
prepare sources icu4c :
  svncheckout icu4c_native : SKIP
  svnupdate icu4c_native : OK
  patch icu4c_native : SKIP
prepare sources libzim :
  gitclone libzim : SKIP
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : SKIP
  gitupdate kiwix-lib : OK
prepare sources kiwix-android :
  gitclone kiwix-android : SKIP
  gitupdate kiwix-android : OK
prepare sources kiwix-android-custom :
  gitclone kiwix-android-custom : SKIP
  gitupdate kiwix-android-custom : OK
[BUILD]
build toolchain android-ndk :
  build_platform android-ndk : OK
  fix_permission_right android-ndk : OK
build toolchain android-sdk :
  build_platform android-sdk : SKIP
  fix_licenses android-sdk : SKIP
build Gradle :
  install Gradle : OK
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build uuid :
  configure uuid : OK
  compile uuid : OK
  install uuid : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c_native :
  configure icu4c_native : OK
  compile icu4c_native : OK
  install icu4c_native : SKIP
build icu4c_cross-compile :
  configure icu4c_cross-compile : OK
  compile icu4c_cross-compile : OK
  install icu4c_cross-compile : OK
build libzim :
  configure libzim : OK
  compile libzim : OK
  install libzim : OK
build kiwix-lib :
  configure kiwix-lib : OK
  compile kiwix-lib : OK
  install kiwix-lib : OK
build kiwix-android :
SKIP
build kiwix-android-custom :
  configure kiwix-android-custom : ERROR
Traceback (most recent call last):
  File "./kiwix-build.py", line 947, in <module>
    builder.run()
  File "./kiwix-build.py", line 885, in run
    self.build()
  File "./kiwix-build.py", line 875, in build
    builder.build()
  File "/home/kelson/kiwix-build/dependencies.py", line 412, in build
    self.command('configure', self._configure)
  File "/home/kelson/kiwix-build/dependency_utils.py", line 206, in command
    return self.target.command(*args, **kwargs)
  File "/home/kelson/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/home/kelson/kiwix-build/dependencies.py", line 441, in _configure
    shutil.copytree(pj(self.buildEnv.install_dir, 'kiwix-lib'), pj(self.build_path, 'kiwixlib', 'src', 'main'))
  File "/home/kelson/kiwix-build/lib/python3.5/shutil.py", line 303, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/home/kelson/kiwix-build/BUILD_android_arm/INSTALL/kiwix-lib'

kiwix-index - where is it now ?

Hi,

I'm trying to compile kiwix for an arm64 system but it seems to be missing kiwix-index ?

I've cloned the repo and built everything but kiwix-index seems to be missing

Many thanks

Unable to compile pugixml on macOS

Running ./kiwix-build.py

prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : ERROR
Traceback (most recent call last):
  File "/Users/kelson/kiwix-build/bin/mesontest", line 19, in <module>
    from mesonbuild import mesonmain
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonmain.py", line 18, in <module>
    from . import environment, interpreter, mesonlib
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/environment.py", line 17, in <module>
    from . import coredata
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/coredata.py", line 20, in <module>
    from .mesonlib import MesonException, commonpath
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonlib.py", line 60, in <module>
    meson_command = python_command + [detect_meson_py_location()]
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonlib.py", line 51, in detect_meson_py_location
    raise RuntimeError('Could not determine how to run Meson. Please file a bug with details.')
RuntimeError: Could not determine how to run Meson. Please file a bug with details.
run command 'mesontest --verbose'
current directory is '/Users/kelson/kiwix-build/BUILD_native_dyn/pugixml-1.2'
env is :
  TERM_PROGRAM : 'Apple_Terminal'
  SHELL : '/bin/bash'
  TERM : 'xterm-256color'
  TMPDIR : '/var/folders/s7/frvr3wkd3_34_kkd353f70k40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.snnquLCEZx/Render'
  TERM_PROGRAM_VERSION : '388.1.1'
  OLDPWD : '/Users/kelson/kiwix-build/ninja'
  TERM_SESSION_ID : '70AA5DAE-B08E-4294-BDBE-01DE72B3E10E'
  USER : 'kelson'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.l21yy24m1l/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/kelson/kiwix-build'
  PATH : '/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/kelson/kiwix-build/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/kelson/kiwix-build'
  XPC_FLAGS : '0x0'
  PS1 : '(kiwix-build) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/kelson'
  LOGNAME : 'kelson'
  LC_CTYPE : 'UTF-8'
  SECURITYSESSIONID : '186a8'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

Compile (and publish as nightly) zim-tools

kiwix-build.py should detect early if a dependency (svn for example) is not available (and complain)

on a fresh new Debian stretch install, I followed the instructions of this readme and Kiwix failed to compile because SVN was missing. it should detect that and just install subversion through apt-get install subversion.

(similarly, why aren't meson and ninja installed that way? just apt-get install meson was enough here...)

macOS: cannot run kiwis-build.py

Hi @mgautierfr, I am trying to run the build script on macOS 10.12.3, with python 3.6, targeting macOS.
I executed sudo python3 kiwix-build.py --build-static build, got the following error message ERROR: kiwix-build is intented to run only on Linux platform
What should I do?

Error building iOS_arm64 on macOS 10.13.2

output:

(venv) Chriss-MBP:kiwix-build chrisli$ ./kiwix-build.py kiwix-lib --target-platform iOS_arm64
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
SKIP
[PREPARE]
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : SKIP
  patch pugixml : SKIP
prepare sources zlib :
  download zlib : SKIP
  extract zlib : SKIP
  patch zlib : SKIP
prepare sources lzma :
  download lzma : SKIP
  extract lzma : SKIP
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : SKIP
prepare sources icu4c :
  svncheckout icu4c_native : SKIP
  svnupdate icu4c_native : OK
  patch icu4c_native : SKIP
prepare sources libzim :
  gitclone libzim : SKIP
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : SKIP
  gitupdate kiwix-lib : OK
[BUILD]
build pugixml :
  configure pugixml : ERROR
Traceback (most recent call last):
  File "./kiwix-build.py", line 1030, in <module>
    builder.run()
  File "./kiwix-build.py", line 964, in run
    self.build()
  File "./kiwix-build.py", line 948, in build
    builder.build()
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 224, in build
    self.command('configure', self._configure)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 219, in command
    return self.target.command(*args, **kwargs)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 346, in _configure
    if not self.target.force_native_build and self.buildEnv.meson_crossfile:
  File "./kiwix-build.py", line 342, in __getattr__
    return getattr(self.options, name)
AttributeError: 'Namespace' object has no attribute 'meson_crossfile'

Please provide a way to clean from a failed, partial run of kiwix-build.py

I remember in Lyon that I had to delete a folder during the early stages of trying to get kiwis-build.py to run successfully to completion. I've run into what seems a similar problem when trying to run kiwis-build.py on a Raspberry Pi (with Python 3.4.3 provided using pyenv)

I've manually deleted various folders, including ARCHIVE, SOURCE and BUILD_native_dyn I've tried to see if there's a --clean option in the script (there doesn't seem to be one), and am now a bit stuck trying to make sense of what's causing the script to first SKIP and then ERROR when trying to obtain xapian-core. This could well be a local issue on my machine (a Raspberry Pi 3 running the latest patches of Raspbian) or perhaps the script isn't yet ready to run on a Pi / build for Pi's.

Anyway, back to the reason for this issue - I'd like a known, reliable way to reset the environment
and also (if practical) some better logging to help me uncover what's been going wrong.

Write nightly builds at the right place

in http://download.kiwix.org/nightly/bin//

Use the provided CMake file to build pugixml.

Pugixml has a Cmake build system who generate a pkg-config file.
We do not need a custom meson file.

See kiwix/libkiwix#49

Error building on macOS 10.13.1

(kiwix-build) Chriss-MacBook-Pro:kiwix-build chrisli$ ./kiwix-build.py --help
usage: kiwix-build.py [-h] [--working-dir WORKING_DIR] [--libprefix LIBPREFIX]
                      [--target-platform {native_dyn,native_static,win32_dyn,win32_static,armhf_dyn,armhf_static,android_arm,android_arm64,android_mips,android_mips64,android_x86,android_x86_64,iOS_arm64}]
                      [--verbose] [--hide-progress] [--skip-source-prepare]
                      [--build-deps-only] [--no-cert-check] [--clean-at-end]
                      [--android-custom-app ANDROID_CUSTOM_APP]
                      [--zim-file-url ZIM_FILE_URL]
                      [--zim-file-size ZIM_FILE_SIZE]
                      [TARGET]

positional arguments:
  TARGET

optional arguments:
  -h, --help            show this help message and exit
  --working-dir WORKING_DIR
  --libprefix LIBPREFIX
  --target-platform {native_dyn,native_static,win32_dyn,win32_static,armhf_dyn,armhf_static,android_arm,android_arm64,android_mips,android_mips64,android_x86,android_x86_64,iOS_arm64}
  --verbose, -v         Print all logs on stdout instead of in specific log
                        files per commands
  --hide-progress       Hide intermediate progress information.
  --skip-source-prepare
                        Skip the source download part
  --build-deps-only     Build only the dependencies of the specified targets.

advanced:
  --no-cert-check       Skip SSL certificate verification during download
  --clean-at-end        Clean all intermediate files after the (successfull)
                        build

custom app:
  Android custom app specific options

  --android-custom-app ANDROID_CUSTOM_APP
                        The custom android app to build
  --zim-file-url ZIM_FILE_URL
                        The url of the zim file to download
  --zim-file-size ZIM_FILE_SIZE
                        The size of the zim file.
(kiwix-build) Chriss-MacBook-Pro:kiwix-build chrisli$ ./kiwix-build.py --target-platform native_dyn
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
 - autoconf : SKIP
 - automake : SKIP
 - libtool : SKIP
 - cmake : SKIP
 - pkg-config : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources pugixml :
  download pugixml : OK1.50%
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK0.41%
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : OK0.02%
  extract lzma : OK
prepare sources xapian-core :
  download xapian-core : OK0.26%
  extract xapian-core : OK
prepare sources icu4c :
  svncheckout icu4c : OK
  svnupdate icu4c : OK
  patch icu4c : OK
prepare sources libzim :
  gitclone libzim : OK
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : OK
  gitupdate kiwix-lib : OK
prepare sources libmicrohttpd :
  download libmicrohttpd : OK0.31%
  extract libmicrohttpd : OK
prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c :
  configure icu4c : OK
  compile icu4c : OK
  install icu4c : OK
build libzim :
  configure libzim : OK
  compile libzim : ERROR
[1/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_part.cpp.o' -MF 'src/zim@sha/file_part.cpp.o.d' -o 'src/zim@sha/file_part.cpp.o' -c ../../SOURCE/libzim/src/file_part.cpp
[2/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/articlesource.cpp.o' -MF 'src/zim@sha/articlesource.cpp.o.d' -o 'src/zim@sha/articlesource.cpp.o' -c ../../SOURCE/libzim/src/articlesource.cpp
[3/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/envvalue.cpp.o' -MF 'src/zim@sha/envvalue.cpp.o.d' -o 'src/zim@sha/envvalue.cpp.o' -c ../../SOURCE/libzim/src/envvalue.cpp
[4/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/fileheader.cpp.o' -MF 'src/zim@sha/fileheader.cpp.o.d' -o 'src/zim@sha/fileheader.cpp.o' -c ../../SOURCE/libzim/src/fileheader.cpp
[5/45] cc  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -std=c11 -O0 -g -pthread -MMD -MQ 'src/zim@sha/md5.c.o' -MF 'src/zim@sha/md5.c.o.d' -o 'src/zim@sha/md5.c.o' -c ../../SOURCE/libzim/src/md5.c
[6/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/dirent.cpp.o' -MF 'src/zim@sha/dirent.cpp.o.d' -o 'src/zim@sha/dirent.cpp.o' -c ../../SOURCE/libzim/src/dirent.cpp
[7/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/cluster.cpp.o' -MF 'src/zim@sha/cluster.cpp.o.d' -o 'src/zim@sha/cluster.cpp.o' -c ../../SOURCE/libzim/src/cluster.cpp
[8/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/article.cpp.o' -MF 'src/zim@sha/article.cpp.o.d' -o 'src/zim@sha/article.cpp.o' -c ../../SOURCE/libzim/src/article.cpp
[9/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_compound.cpp.o' -MF 'src/zim@sha/file_compound.cpp.o.d' -o 'src/zim@sha/file_compound.cpp.o' -c ../../SOURCE/libzim/src/file_compound.cpp
[10/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_reader.cpp.o' -MF 'src/zim@sha/file_reader.cpp.o.d' -o 'src/zim@sha/file_reader.cpp.o' -c ../../SOURCE/libzim/src/file_reader.cpp
FAILED: src/zim@sha/file_reader.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_reader.cpp.o' -MF 'src/zim@sha/file_reader.cpp.o.d' -o 'src/zim@sha/file_reader.cpp.o' -c ../../SOURCE/libzim/src/file_reader.cpp
../../SOURCE/libzim/src/file_reader.cpp:128:12: error: moving a local object in a return statement prevents copy elision [-Werror,-Wpessimizing-move]
    return std::move(buffer);
           ^
../../SOURCE/libzim/src/file_reader.cpp:128:12: note: remove std::move call here
    return std::move(buffer);
           ^~~~~~~~~~      ~
../../SOURCE/libzim/src/file_reader.cpp:138:12: error: moving a local object in a return statement prevents copy elision [-Werror,-Wpessimizing-move]
    return std::move(ret_buffer);
           ^
../../SOURCE/libzim/src/file_reader.cpp:138:12: note: remove std::move call here
    return std::move(ret_buffer);
           ^~~~~~~~~~          ~
2 errors generated.
[11/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file.cpp.o' -MF 'src/zim@sha/file.cpp.o.d' -o 'src/zim@sha/file.cpp.o' -c ../../SOURCE/libzim/src/file.cpp
[12/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/blob.cpp.o' -MF 'src/zim@sha/blob.cpp.o.d' -o 'src/zim@sha/blob.cpp.o' -c ../../SOURCE/libzim/src/blob.cpp
[13/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/template.cpp.o' -MF 'src/zim@sha/template.cpp.o.d' -o 'src/zim@sha/template.cpp.o' -c ../../SOURCE/libzim/src/template.cpp
[14/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/md5stream.cpp.o' -MF 'src/zim@sha/md5stream.cpp.o.d' -o 'src/zim@sha/md5stream.cpp.o' -c ../../SOURCE/libzim/src/md5stream.cpp
[15/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/buffer.cpp.o' -MF 'src/zim@sha/buffer.cpp.o.d' -o 'src/zim@sha/buffer.cpp.o' -c ../../SOURCE/libzim/src/buffer.cpp
[16/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/uuid.cpp.o' -MF 'src/zim@sha/uuid.cpp.o.d' -o 'src/zim@sha/uuid.cpp.o' -c ../../SOURCE/libzim/src/uuid.cpp
[17/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search_iterator.cpp.o' -MF 'src/zim@sha/search_iterator.cpp.o.d' -o 'src/zim@sha/search_iterator.cpp.o' -c ../../SOURCE/libzim/src/search_iterator.cpp
FAILED: src/zim@sha/search_iterator.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search_iterator.cpp.o' -MF 'src/zim@sha/search_iterator.cpp.o.d' -o 'src/zim@sha/search_iterator.cpp.o' -c ../../SOURCE/libzim/src/search_iterator.cpp
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:20:
../../SOURCE/libzim/src/xapian/myhtmlparse.h:45:7: error: 'zim::MyHtmlParser::parse_html' hides overloaded virtual function [-Werror,-Woverloaded-virtual]
        void parse_html(const string &text, const string &charset_,
             ^
../../SOURCE/libzim/src/xapian/htmlparse.h:46:15: note: hidden overloaded virtual function 'zim::HtmlParser::parse_html' declared here: different number of parameters (1 vs 3)
        virtual void parse_html(const string &text);
                     ^
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:24:
../../SOURCE/libzim/src/search_internal.h:31:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct Search::InternalData {
^
../../SOURCE/libzim/include/zim/search.h:58:10: note: did you mean struct here?
         class InternalData;
         ^~~~~
         struct
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:24:
../../SOURCE/libzim/src/search_internal.h:39:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct search_iterator::InternalData {
^
../../SOURCE/libzim/include/zim/search_iterator.h:60:9: note: did you mean struct here?
        class InternalData;
        ^~~~~
        struct
3 errors generated.
[18/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/fileimpl.cpp.o' -MF 'src/zim@sha/fileimpl.cpp.o.d' -o 'src/zim@sha/fileimpl.cpp.o' -c ../../SOURCE/libzim/src/fileimpl.cpp
[19/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search.cpp.o' -MF 'src/zim@sha/search.cpp.o.d' -o 'src/zim@sha/search.cpp.o' -c ../../SOURCE/libzim/src/search.cpp
FAILED: src/zim@sha/search.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search.cpp.o' -MF 'src/zim@sha/search.cpp.o.d' -o 'src/zim@sha/search.cpp.o' -c ../../SOURCE/libzim/src/search.cpp
In file included from ../../SOURCE/libzim/src/search.cpp:22:
../../SOURCE/libzim/src/search_internal.h:31:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct Search::InternalData {
^
../../SOURCE/libzim/include/zim/search.h:58:10: note: did you mean struct here?
         class InternalData;
         ^~~~~
         struct
In file included from ../../SOURCE/libzim/src/search.cpp:22:
../../SOURCE/libzim/src/search_internal.h:39:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct search_iterator::InternalData {
^
../../SOURCE/libzim/include/zim/search_iterator.h:60:9: note: did you mean struct here?
        class InternalData;
        ^~~~~
        struct
2 errors generated.
ninja: build stopped: subcommand failed.
run command 'ninja -v'
current directory is '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/libzim'
env is :
  TERM_PROGRAM : 'iTerm.app'
  TERM : 'xterm-256color'
  SHELL : '/bin/bash'
  TMPDIR : '/var/folders/w0/ph55y7r54tld0qb0kyybvbs40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.nrCFYuutOL/Render'
  TERM_PROGRAM_VERSION : '3.1.5'
  OLDPWD : '/Users/chrisli/Developer/kiwix-build/ninja'
  TERM_SESSION_ID : 'w0t0p0:5C85ED98-7F48-42D8-8599-4475C4AAE225'
  USER : 'chrisli'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.XjJNJcWRrR/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/chrisli/Developer/kiwix-build'
  PATH : '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/chrisli/Developer/kiwix-build/bin:/Users/chrisli/.cargo/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/chrisli/Developer/kiwix-build'
  LANG : 'en_US.UTF-8'
  ITERM_PROFILE : 'Default'
  XPC_FLAGS : '0x0'
  PS1 : '(kiwix-build) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/chrisli'
  COLORFGBG : '15;0'
  ITERM_SESSION_ID : 'w0t0p0:5C85ED98-7F48-42D8-8599-4475C4AAE225'
  LOGNAME : 'chrisli'
  COLORTERM : 'truecolor'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

Error building native_dyn on macOS 10.13.2

output

(venv) Chriss-MBP:kiwix-build chrisli$ ./kiwix-build.py --target-platform native_dyn
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
 - autoconf : SKIP
 - automake : SKIP
 - libtool : SKIP
 - cmake : SKIP
 - pkg-config : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources pugixml :
  download pugixml : OK1.50%
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK0.41%
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : OK0.02%
  extract lzma : OK
prepare sources xapian-core :
  download xapian-core : OK0.26%
  extract xapian-core : OK
prepare sources icu4c :
  svncheckout icu4c : OK
  svnupdate icu4c : OK
  patch icu4c : OK
prepare sources libzim :
  gitclone libzim : OK
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : OK
  gitupdate kiwix-lib : OK
prepare sources libmicrohttpd :
  download libmicrohttpd : OK0.31%
  extract libmicrohttpd : OK
prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c :
  configure icu4c : OK
  compile icu4c : OK
  install icu4c : OK
build libzim :
  configure libzim : OK
  compile libzim : OK
  test libzim : OK
  install libzim : OK
build kiwix-lib :
  configure kiwix-lib : OK
  compile kiwix-lib : OK
  test kiwix-lib : OK
  install kiwix-lib : OK
build libmicrohttpd :
  configure libmicrohttpd : OK
  compile libmicrohttpd : OK
  install libmicrohttpd : OK
build kiwix-tools :
  configure kiwix-tools : OK
  compile kiwix-tools : ERROR
[1/13] c++  -Isrc/reader/kiwix-read@exe -Isrc/reader -I../../SOURCE/kiwix-tools/src/reader -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -MF 'src/reader/kiwix-read@exe/kiwix-read.cpp.o.d' -o 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -c ../../SOURCE/kiwix-tools/src/reader/kiwix-read.cpp
../../SOURCE/kiwix-tools/src/reader/kiwix-read.cpp:85:18: warning: unused variable 'contentLength' [-Wunused-variable]
    unsigned int contentLength = 0;
                 ^
1 warning generated.
[2/13] c++  -Isrc/searcher/kiwix-search@exe -Isrc/searcher -I../../SOURCE/kiwix-tools/src/searcher -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -MF 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o.d' -o 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -c ../../SOURCE/kiwix-tools/src/searcher/kiwix-search.cpp
[3/13] c++  -o src/reader/kiwix-read 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[4/13] /Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin/kiwix-compile-resources --cxxfile static/server/server-resources.cpp --hfile static/server/server-resources.h ../../SOURCE/kiwix-tools/static/server/resources_list.txt
jquery-ui/jquery-ui.structure.min.css ['jquery_ui', 'jquery_ui_structure_min_css']
jquery-ui/jquery-ui.min.js ['jquery_ui', 'jquery_ui_min_js']
jquery-ui/external/jquery/jquery.js ['jquery_ui', 'external', 'jquery', 'jquery_js']
jquery-ui/images/ui-bg_flat_0_aaaaaa_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_0_aaaaaa_40x100_png']
jquery-ui/images/ui-bg_flat_75_ffffff_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_75_ffffff_40x100_png']
jquery-ui/images/ui-icons_222222_256x240.png ['jquery_ui', 'images', 'ui_icons_222222_256x240_png']
jquery-ui/images/ui-bg_glass_55_fbf9ee_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_55_fbf9ee_1x400_png']
jquery-ui/images/ui-bg_highlight-soft_75_cccccc_1x100.png ['jquery_ui', 'images', 'ui_bg_highlight_soft_75_cccccc_1x100_png']
jquery-ui/images/ui-bg_glass_65_ffffff_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_65_ffffff_1x400_png']
jquery-ui/images/ui-icons_2e83ff_256x240.png ['jquery_ui', 'images', 'ui_icons_2e83ff_256x240_png']
jquery-ui/images/ui-icons_cd0a0a_256x240.png ['jquery_ui', 'images', 'ui_icons_cd0a0a_256x240_png']
jquery-ui/images/ui-icons_888888_256x240.png ['jquery_ui', 'images', 'ui_icons_888888_256x240_png']
jquery-ui/images/ui-bg_glass_75_e6e6e6_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_e6e6e6_1x400_png']
jquery-ui/images/animated-overlay.gif ['jquery_ui', 'images', 'animated_overlay_gif']
jquery-ui/images/ui-bg_glass_75_dadada_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_dadada_1x400_png']
jquery-ui/images/ui-icons_454545_256x240.png ['jquery_ui', 'images', 'ui_icons_454545_256x240_png']
jquery-ui/images/ui-bg_glass_95_fef1ec_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_95_fef1ec_1x400_png']
jquery-ui/jquery-ui.theme.min.css ['jquery_ui', 'jquery_ui_theme_min_css']
jquery-ui/jquery-ui.min.css ['jquery_ui', 'jquery_ui_min_css']
home.html.tmpl ['home_html_tmpl']
include.html.part ['include_html_part']
taskbar.css ['taskbar_css']
taskbar.html.part ['taskbar_html_part']
global_taskbar.html.part ['global_taskbar_html_part']
[5/13] c++  -o src/searcher/kiwix-search 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[6/13] c++  -Isrc/installer/kiwix-install@exe -Isrc/installer -I../../SOURCE/kiwix-tools/src/installer -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -MF 'src/installer/kiwix-install@exe/kiwix-install.cpp.o.d' -o 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -c ../../SOURCE/kiwix-tools/src/installer/kiwix-install.cpp
[7/13] c++  -Isrc/manager/kiwix-manage@exe -Isrc/manager -I../../SOURCE/kiwix-tools/src/manager -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -MF 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o.d' -o 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -c ../../SOURCE/kiwix-tools/src/manager/kiwix-manage.cpp
[8/13] c++  -o src/installer/kiwix-install 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[9/13] c++  -o src/manager/kiwix-manage 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[10/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/request_context.cpp.o' -MF 'src/server/kiwix-serve@exe/request_context.cpp.o.d' -o 'src/server/kiwix-serve@exe/request_context.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/request_context.cpp
[11/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -MF 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o.d' -o 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp
FAILED: src/server/kiwix-serve@exe/kiwix-serve.cpp.o
c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -MF 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o.d' -o 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:111:11: warning: comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Wtautological-constant-out-of-range-compare]
  if (pos != std::string::npos) {
      ~~~ ^  ~~~~~~~~~~~~~~~~~
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:221:35: warning: 'MHD_create_response_from_data' is deprecated: MHD_create_response_from_data() is deprecated, use MHD_create_response_from_buffer() [-Wdeprecated-declarations]
  struct MHD_Response* response = MHD_create_response_from_data(
                                  ^
/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/microhttpd.h:2032:1: note: 'MHD_create_response_from_data' has been explicitly marked deprecated here
_MHD_DEPR_FUNC("MHD_create_response_from_data() is deprecated, use MHD_create_response_from_buffer()") \
^
/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/microhttpd.h:244:44: note: expanded from macro '_MHD_DEPR_FUNC'
#define _MHD_DEPR_FUNC(msg) __attribute__((deprecated(msg)))
                                           ^
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:547:27: error: no member named 'getHtml' in 'kiwix::Searcher'
      content = searcher->getHtml();
                ~~~~~~~~  ^
2 warnings and 1 error generated.
[12/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o' -MF 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o.d' -o 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o' -c static/server/server-resources.cpp
ninja: build stopped: subcommand failed.
run command 'ninja -v'
current directory is '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/kiwix-tools'
env is :
  TERM_PROGRAM : 'iTerm.app'
  TERM : 'xterm-256color'
  SHELL : '/bin/bash'
  TMPDIR : '/var/folders/w0/ph55y7r54tld0qb0kyybvbs40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.gRTBqiVMvv/Render'
  TERM_PROGRAM_VERSION : '3.1.5'
  OLDPWD : '/Users/chrisli/Developer/kiwix-build/ninja'
  TERM_SESSION_ID : 'w0t2p0:CF89E50E-4DF2-4263-94D4-A20B4022D45F'
  USER : 'chrisli'
  COMMAND_MODE : 'unix2003'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.4Ly3PHQaTC/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/chrisli/Developer/kiwix-build/venv'
  PATH : '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/chrisli/Developer/kiwix-build/venv/bin:/Users/chrisli/.cargo/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/chrisli/Developer/kiwix-build'
  LANG : 'en_US.UTF-8'
  ITERM_PROFILE : 'Default'
  XPC_FLAGS : '0x0'
  PS1 : '(venv) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/chrisli'
  COLORFGBG : '0;15'
  ITERM_SESSION_ID : 'w0t2p0:CF89E50E-4DF2-4263-94D4-A20B4022D45F'
  LOGNAME : 'chrisli'
  SECURITYSESSIONID : '186a6'
  COLORTERM : 'truecolor'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

Add Linux/ARM to CI build targets

Please include support for arm, especially rpi in builds

Compile (and publish as nightly) zimwriterfs

Nightly Latest folder empty or has unique names

The nightly folder at http://download.kiwix.org/nightly/latest/ is empty a lot of times. When the folder is not empty, the filenames can be unique, causing scripts or docker builds to fail.

If the names in nightly/latest/ are always the same, then we can always point our software towards the latest build. Which is most helpful when we are all submitting bug reports.

desired:
http://download.kiwix.org/nightly/latest/kiwix-tools_linux64.tar.gz

actual:
http://download.kiwix.org/nightly/latest/kiwix-tools_linux64_2017-11-12.tar.gz
or sometimes blank:
http://download.kiwix.org/nightly/latest/

Improve kiwix-build README

README should clearly explain the compilation procedure. This means step-by-step procedures which can be follow blindly with short explanation about what is going on:

• What are the dependencies (with minimal version)
• How to compile kiwix-lib and kiwix-tools under Linux static/dynamic by downloading locally deps
• How to cross-compile kiwix-lib and kiwix-tools under Linux for Windows static/dynamic by downloading locally deps

CI kiwix-build basic integration with Travis

On each commit the CI should build:

• On Ubuntu, compile statically kiwix-lib with local deps (downloaded)
• On Ubuntu, compile statically kiwix-tools with local deps (downloaded)
• On Ubuntu, cross-compile for windows statically local deps (downloaded)

How should I implement iOS sdk Toolchain if no download is necessary

As the discussion between @kelson42 and me, it might be a good idea to simultaneously start cross-platform build development for iOS. So in branch apple_platforms, I have added iOS SDK class

class iOS_sdk(Toolchain):
    pass

But different from android, kiwix-build do not need to download the SDK, since it is already part of Xcode.app. I just need to run xcodebuild -version -sdk iphonesimulator | grep -E '^Path' | sed 's/Path: //' to get the SDK path. How should I implement this class?

Modify the code to work on Jessie (python 3.4.2 and context argument)

The default python3 on Raspbian Jessie is 3.4.2

The current code uses 'context' which is first available in Python 3.4.3

Let's see if we can find a way to work on 3.4.2 so kiwix-build can run on Raspbian Jessie.

Dependence between kiwix and openzim.

The recent PR #21 allow kiwix-build to build the zimwriterfs project.
This add no extra dependencies between the kiwix projects and zimwriterfs. It just said to kiwix-build how to build zimwriterfs.

This PR leads to a discussion between @kelson42 and me about what is the difference between the two organization openzim and kiwix.
I will try to be objective here, but I may not succeed, I'm part of the discussion. @kelson42, do not hesitate to correct or complete me.

The main argument of @kelson42 is that the openzim and kiwix are two different projects and must stay as it. People should be able to use zim (through zimlib, zimwriterfs or any other tools) without using at all kiwix or kiwix-related projects. Thus, he asks me to think of how we could separate the two projects and do not add zimwriterfs to kiwix-build. (But we agree that for now, it worth to accept the PR as we could easily remove zimwriterfs or split/duplicate kiwix-build if necessary)

I globally agree with this will. The zim format is a (somehow advanced) archive format and should stay as it. Kiwix is a application that use the zim format to store and display web content (mainly mediawiki content, but not only)

But, saying that, I don't see why the kiwix team couldn't have tools to use and compile zimwriterfs.
We already have tool to compile other dependencies as xapian, pugixml, .. and zimlib.
zimwriterfs is not intended to be used by kiwix end user, but I don't see why we (kiwix) should avoid publish a tool to compile it. If someone asks about it, we just could answer «At kiwix, we build zimwriterfs this way. It simpler this way, but do as you want. Use it, or not, we don't care.»
We already greatly use zimlib (kiwix/kiwix-lib and thus kiwix/kiwix-tools, kiwix/kiwix-android, kiwix/kiwix-IOS, kiwix/node-libzim) and zimwriterfs (kiwix/mwoffliner, kiwix/sotoki, kiwix/phet, kiwix/youtube, kiwix/TED, kiwix/gutemberg). And I'm not aware of other projects using libzim.

The other solutions are to duplicate kiwix-build or move kiwix-build to a generic and independent builder. And I'm appealing by none of them.

Make CI temporary dependencies available through HTTP

These dependencies are necessary to compile many repositories without having each time to recompile everything and even more important without having to use kiwix-build.

This dependencies needs to be also available to everybody, otherwise the CI won't be able to test merge requests hosted on external git repositories.

Integrate post-command `change_install_name` for movable kiwix-lib on mac_os

To be able to move library on mac_os we need to change the post script to change the install_name "inside" the binary.

Something's awry with downloading xapian-core in kiwix-build

This may just be me / my environment however I've had problems caused by what seem to be issues either downloading or extracting the xapian-core dependency as part of kiwix-build.py

Here's the error trace when trying to build the arm branch. It also occurs on the same device (a Pi 3) when trying to build from the master branch

I'll try to run the build in another linux environment to triangulate the scope of the problem.

(kiwix)pi@rachel ~/kiwix-dev/kiwix-build $ pyenv exec python kiwix-build.py --target-platform raspberrypi_dyn
[INSTALL PACKAGES]
 - cmake : SKIP
 - automake : SKIP
 - ccache : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources for toolchain armhf :
  gitclone armhf : OK
  gitupdate armhf : OK
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : SKIP
  extract lzma : OK
prepare sources uuid :
  download uuid : OK
  extract uuid : OK
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : ERROR
Traceback (most recent call last):
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 75, in extract_archive
    archive = tarfile.open(archive_path)
  File "/home/pi/.pyenv/versions/3.4.3/lib/python3.4/tarfile.py", line 1553, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kiwix-build.py", line 862, in <module>
    builder.run()
  File "kiwix-build.py", line 831, in run
    self.prepare_sources()
  File "kiwix-build.py", line 812, in prepare_sources
    source.prepare()
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 117, in prepare
    self.command('extract', self._extract)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 86, in command
    return self.target.command(*args, **kwargs)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 106, in _extract
    name=self.source_dir)
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 115, in extract_archive
    if archive is not None:
UnboundLocalError: local variable 'archive' referenced before assignment

kiwix-build install dir should be customizable

The scripts should allow to:

• specify the install directory on the command line (necessary for the packaging)
• run the install separatly from the compilation

macOS: error when build kiwix-lib natively

Using code from branch apple-platforms, building on macOS natively, encountered errors when building kiwix-lib. Please see log for detail.
log.txt