phusion/holy-build-box

System for building cross-distribution Linux binaries

System for building cross-distribution Linux binaries

Holy Build Box is a system for building "portable" binaries for Linux: binaries that work on pretty much any Linux distribution. This works by providing an easy-to-use compilation environment with an old glibc version. Holy Build Box can produce amd64 and arm64 binaries.

Resources:: Website | Issue tracker

Table of contents

• Problem introduction
  • Glibc symbols
  • Why statically linking to glibc is a bad idea
  • Dynamic libraries
• Features
• Featured use cases
• Getting started
  • Tutorials
  • Guides
  • Caveats
• FAQ
  • Who should be interested in portable Linux binaries?
  • Which operating systems does Holy Build Box support?
  • Which Linux distributions do binaries compiled with Holy Build Box support?
  • Static linking introduces security problems. How do you deal with this?
  • How does Holy Build Box compare to using Docker to package up an application?
  • How does Holy Build Box compare to Go?
  • Is Holy Build Box suitable for all applications?
  • How should I deal with interpreted applications, such as ones written in Ruby, Python or Node.js?
  • Why the name "Holy Build Box"?
• Other comparable systems

---

Problem introduction

If you have ever tried to build a C/C++ Linux binary, then you will probably have noticed that it may not run on other Linux distributions, or even other versions of the same Linux distribution. This happens even if your application does not use newer APIs. This is in stark contrast to Windows binaries, which tend to on pretty much every Windows machine. There are many reasons why this is so. This section introduces you to each problem, and whether and how Holy Build Box solves that problem.

Glibc symbols

The most prominent reason why binaries aren't portable is glibc symbols. When you try to run an older Linux distribution a binary that is compiled on a newer Linux distribution, you may see an error message like this:

./foo: /lib/libc.so.6: version `GLIBC_2.11' not found

Each function in glibc -- each symbol -- actually has multiple versions. This allows the glibc developers to change the behavior of a function without breaking backwards compatibility with applications that happen to rely on bugs or implementation-specific behavior. During the linking phase, the linker "helpfully" links against the most recent version of the symbol. The thing is, glibc introduces new symbol versions very often, resulting in binaries that will most likely depend on a recent glibc.

The only way to tell the compiler and linker to use older symbol versions is by using linker scripts. However, this requires you to specify the version for each and every symbol, which is an undoable task.

Holy Build Box solves the glibc symbol problem by providing a tightly-controlled build environment that contains an old version of glibc.

Why statically linking to glibc is a bad idea

The glibc symbol problem can be solved by statically linking to glibc. However, this is not the approach that Holy Build Box advocates. Statically linking to glibc wastes a lot of space. A simple 200 KB program can suddenly become 5 MB.

Statically linking to glibc also introduces various runtime problems. For example, such binaries are not able to call dlopen(), so applications which want to load plugins at runtime aren't compatible with this approach.

Dynamic libraries

Another prominent reason why binaries aren't portable is dynamic libraries. Any non-trivial application has dependencies, but every distribution has different library versions.

It is awkward to ship a binary to users, only for them to discover that they need to install some library first before your binary works. Even if your user accepts that they need to install some library, that still does not fully solve the problem, because newer distributions may ship a newer yet binary-incompatible version of a dependency library. So just compiling your application on an older distribution is not enough.

The solution is to statically link to dependencies. Note that we are not advocating static linking of everything: the Holy Build Box approach is to statically link to all dependencies, except for glibc and other system libraries that are found on pretty much every Linux distribution, such as libpthread and libm.

There are some problems with static linking though. Performing static linking as we advocate it, is very awkward on most systems for the following reasons:

1. Many build system tools are only designed with dynamic linking in mind, and require many tweaks before they work with static linking.

   Example: the default OpenSSL pkg-config entries do not work with static linking at all.

2. If the build environment is not tightly controlled, then various build systems may pull in too many dependencies.

   Example: suppose that you have installed libssh on your workstation because some utility application that you use requires it. Now suppose that you are trying to produce a portable binary of some third-party application. That application's build system automatically detects whether libssh is installed, and if so, links to it. However, you do not want that application to have SSH functionality. To produce the desired binary, you now need to uninstall libssh, thereby breaking the utility application that you are using.

3. The static libraries provided by your distribution may not be compiled with the flags you want.

Holy Build Box solves problem #2 by providing a tightly-controlled build environment that is isolated from your host environment. This works through Docker: Holy Build Box is a Docker image.

Holy Build Box partially solves problem #1 and #3 by providing static versions of often-used libraries that tend to be a pain to set up for proper static linking. Holy Build Box even provides multiple versions of these libraries, compiled with different compilation flags.

Features

Isolated build environment based on Docker and CentOS 7

The Holy Build Box environment is built on CentOS 7. This allows it to produce binaries that work on pretty much any amd64 or arm64 Linux distribution released since 2015. The only currently-prominent Linux distribution that the produced binaries may not run on, is Alpine Linux. See the FAQ entry.

The environment is bare-bones with almost nothing installed. Besides the basics, only a compiler toolchain is provided. The toolchain is more recent than the one provided by CentOS 7.

• GCC 9.3.1 (C and C++ support; in particular, C++14 is supported)
• GNU make
• autoconf 2.69
• automake 1.13.4
• libtool 2.4.2
• pkg-config 0.29.2
• ccache 3.7.12
• CMake 3.19.3
• Python 2.7.5 (+ setuptools and pip)

Included static libraries

Holy Build Box also includes static versions of certain libraries. These libraries are more recent than the ones shipped with CentOS 7.

• zlib 1.2.11
• OpenSSL 1.1.1i
• curl and libcurl 7.74.0
• SQLite 2020-3340000

These libraries are provided in multiple variants, each compiled with different compilation flags. The different variants will be covered with in Tutorial 5: Using library variants and in the Library variants guide.

Security hardening

Holy Build Box makes it easy to compile your application with special security hardening features:

• Protection against stack overflows and stack smashing
• Extra bounds checking in common functions
• Load time address randomization
• Read-only global offset table

This is covered in Tutorial 5: Library variants and in the Security hardening binaries.

Featured use cases

Kallisto bioinformatics research software

Pall Melsted, part of a team of researchers from the University of Iceland and the University of California Berkeley, uses Holy Build Box to build binaries for Kallisto, their RNA transcript quantification software.

Their research software is meant to be used by other academics, so Melsted has no control over which systems their software is used on. These other academics are primarily researchers and have few system administration skills.

Holy Build Box allows Melsted's team to publish binaries that other researchers can use with few hassles. Other researchers do not have to worry about upgrading their OS, installing a new compiler or installing dependencies.

At the same time, Melsted's team can pick their favorite and most productive programming language, C++11, without worrying about which Linux systems their users use. Melsted's team also does not have to think about publishing different binaries/packages for all the different Linux systems their users may be using.

Traveling Ruby

Traveling Ruby is a system for producing self-contained Ruby application packages for Windows, Linux and macOS. You could think of it as the Holy Build Box equivalent for Ruby applications.

Traveling Ruby works by providing precompiled Ruby binaries that work on any Linux system. Ruby developers can then take our binaries, bundle it with their application code, and distribute the result as a single zip or tar.gz file to end users. This way, end users do not have to install Ruby in order to use their applications.

The Linux binaries published by the Traveling Ruby project are produced with Holy Build Box.

Passenger web app server

Phusion Passenger is a web app server that lets you launch, operate, optimize and protect web apps, microservices & APIs. It aims to be as easy to use as possible, meaning that it has to be easy to install as well.

Passenger provides Debian and RPM packages which covers most of their users. However there still remain a significant amount of users that are not able to use Debian and RPM packages, but that still want or need binaries. One such user group is Heroku users: it is not feasible to compile Passenger on Heroku. So Heroku users use the precompiled Passenger binaries instead.

Getting started

Tutorials:

• Tutorial 1: Basics
• Tutorial 2: Compilation via a script
• Tutorial 3: Static linking to dependencies
• Tutorial 4: Tweaking the application's build system
• Tutorial 5: Using library variants
• Tutorial 6: Introducing additional static libraries
• Tutorial 7: Verifying binary portability with libcheck

Guides:

• Environment structure
• Library variants
• Installing additional dependencies
• Which system libraries are considered essential?
• Securing the build environment
• Security hardening binaries
• Caching compilation results with ccache

Caveats:

• Libcurl SSL certificate authorities
• Linking C++ applications and libraries

FAQ

Who should be interested in portable Linux binaries?

Holy Build Box is made for developers that fit the following criteria:

• You want to distribute binaries for your C/C++ applications to end users.
• There are many people in your userbase with low to medium system administration skill, and you want your users to be able to easily use your applications, so you want to avoid asking them to compile from source.
• You are wary of having to invest a disproportionate amount of time into building platform-specific packages. You want something that works for the majority of your users, without having to invest too much time.
• You want your users to be easily able to use the latest versions of your software, so waiting for distributions to package your application is not an option.

If you identify with all of the above points, then Holy Build Box is for you.

Intepreted applications whose interpreters are written in C or C++ -- such as Ruby, Python or Node.js -- are also supported, though indirectly.

Some non-developers (i.e. users) may object to the idea of distributing portable binaries. A common objecting is as follows:

"Applications should be packaged through the distribution's native packaging system. Everything else is either broken, inconvenient or insecure."

Packaging applications using distributions' native packaging systems requires a lot of investment from the developer, both in time and in system resources. Many of them are poorly documented. For example, documentation on making Debian packages is hard to find, scattered all over the place, is poorly-written, it often outdated and is hard to understand. On top of that, complying to distributions' packaging guidelines is a lot of work, even more than learning about the packaging system in the first place.

If you are a purist, then you may not care how much time a developer has to invest. You may believe that your native packaging system is the only correct way. That is a valid stand point, but please consider that this stand point may not seem so reasonable to developers. Developers have to target a lot more users than just you. The task soon becomes unwieldly for the developer when they have to invest a huge amount of time into 3 different packaging systems and 12 different distribution versions. It is also not fun for the developer.

We at Phusion spent 3 months trying to package the Passenger application server for Debian and Ubuntu. We spent another 2 months doing the same for Red Hat distributions. Altogether, we spent 5 months during which we did not fix a single bug or introduce a single feature.

We think that distributing portable binaries is a reasonable tradeoff. For all but the most hardcore system administrators who insist on native packages, portable binaries is good enough. Hardcore system administrators are of course still free to wait until their favorite distribution has packaged the application.

Which operating systems does Holy Build Box support?

Holy Build Box only supports amd64 and arm64 Linux.

macOS is not supported. Windows is not supported. Other Unices are not supported. Other CPU architectures, such as 32 bit x86, are not supported.

Which Linux distributions do binaries compiled with Holy Build Box support?

Binaries work on pretty much any glibc-2.17-or-later-based amd64 or arm64 Linux distribution released since approx 2015. A non-exhaustive list:

• Debian >= 8
• Ubuntu >= 14.06
• Red Hat Enterprise Linux >= 7
• CentOS >= 7

Note the mention "glibc-based". Binaries may not be compatible with Linux distributions not based on glibc. For example, early versions of Alpine Linux were based on uclibc, which was not compatible with glibc at all. Later versions of Alpine Linux are based on musl, which is somewhat compatible with glibc. So produced binaries may or may not run on later versions of Alpine Linux.

Static linking introduces security problems. How do you deal with this?

We update the library versions in Holy Build Box regularly. However, application developers will need to recompile their applications on a regular basis too.

There is no way to automatically update application dependencies without recompilation, while also ensuring that their binaries are portable. The two things are mutually incompatible. There is no solution to this. (Having said that, the Meson build system follows an interesting approach which would allow you to at least automatically find out whether a binary is vulnerable.)

If you are an application developer, you should consider the tradeoff. Do your users like having a binary that Just Works(tm)? Do they like this enough that won't mind that an apt-get upgrade won't patch security issues in your application's dependencies, and that they need to wait for an update from you? Are you committed to checking your dependencies' security status and updating your binaries regularly?

How does Holy Build Box compare to using Docker to package up an application?

Docker also solves the portability problem, but it gives the application a very different feel. The application is no longer just a binary. Users will have to install Docker and will have to learn how to use Docker commands in order to use your application. Your users may not particularly care about Docker: maybe they just want to use your application without having to learn about anything else.

Docker images are much larger than binaries produced by Holy Build Box. Docker images contain entire Linux distributions and weigh in the order of hundreds of MB in the average case, or tens of MB if you really did your best to optimize things. Binaries produced by Holy Build Box can be just a few KBs.

On the other hand, compiling an application using Holy Build Box requires advanced knowledge on the C/C++ compiler toolchain. You will regularly run into situations where you need to tweak the build system a little bit before the application properly compiles with static linking. If you are not skilled at using the C/C++ compiler toolchain, then using Docker is easier because it is much more "fire and forget".

How does Holy Build Box compare to Go?

The Go compiler also produces portable Linux binaries. Holy Build Box is meant for existing C/C++ applications. In general, Go is a more productive language than C/C++. So if you are writing a new application, then using Go is an excellent choice. Otherwise, Holy Build Box is for you.

For example, the main reason why we made Holy Build Box was to be able to produce portable binaries for the Phusion Passenger application server and Traveling Ruby. Passenger is a mature codebase written in C++ so we can't just change it entirely to Go. Likewise, Ruby is written in C.

Is Holy Build Box suitable for all applications?

No. Holy Build Box is mainly designed to compile headless applications such as CLI tools, servers and daemons. For example: Phusion Passenger, Nginx, Traveling Ruby.

Supporting graphical applications such as those based on GTK, Qt, SDL, OpenGL etc is outside the scope of this project.

This is however not a technical limitation, but merely a focus one. We, the Holy Build Box maintainers, have no interest in spending time to support graphical applications, but we welcome contributors who would like to take on this challenge. Contact us if you are interested, or submit a pull request.

How should I deal with interpreted applications, such as ones written in Ruby, Python or Node.js?

We recommend that you compile the interpreter with Holy Build Box, and that you package both the interpreter and the application in a single package. If your application makes use of any interpreter extensions, then you should compile those with Holy Build Box too.

We have specific recommendations for some languages:

Ruby

Take a look at Traveling Ruby. The approach taken by Traveling Ruby is exactly what we recommended. Traveling Ruby uses Holy Build Box to build its Linux binaries.

Node.js

If your application has no dependencies on NPM modules with native extensions, then there is no need to use Holy Build Box. You can just download a Node.js binary from www.nodejs.org and package it together with your application and your npm_modules directory. The binaries shipped by www.nodejs.org are already portable and work across Linux distributions.

If your application has a dependency on an NPM module with native extension, either directly or indirectly, then you should compile Node.js and your NPM modules with Holy Build Box. When done, package the compiled Node.js, npm_modules directory and your application.

See also pkg which can bundle Node.js applications into a single portable binary.

Why the name "Holy Build Box"?

Around 2004, I (Hongli Lai) participated in a now-defunct open source project called Autopackage. Back then we were in the middle of the Linux-on-the-desktop hype. One of the things people complained most about was software installation. Every distribution had its own way of installing things, and binaries compiled on one distribution doesn't work on another. This was considered problematic because, as a developer, it is so painful to distribute software to end users. See also the FAQ entry Who should be interested in portable Linux binaries?.

This problem actually still exists -- it was never resolved.

Anyway, the Autopackage project sought to solve this problem by introducing a cross-distribution package manager. We soon realized that solving the packaging aspect only partially solved the problem. Regardless of the packaging, the binaries that were packaged still need to work across Linux distributions.

We saw that Mozilla was able to produce Firefox binaries that work on all Linux distributions. We found out that they did that by constructing a server with a special, tightly-controlled environment that contained an old glibc.

Back in 2004, virtualization was almost non-existent. All Autopackage team members were either in high school or in college. We were only barely able to afford our computers. Constructing a special server just for the purpose of compiling portable binaries was expensive: it meant buying a new computer. The entire idea of constructing such a server was so over our heads that we named Mozilla's build server a "holy build box".

We wanted to give developers a way to produce portable binaries without asking them to construct a holy build box. Virtualization was not a realistic option back then, and we could not imagine that many people would want to buy a new computer only for the purpose of producing portable binaries, especially seeing that Windows developers didn't have to do that either. So we wrote a set of scripts which automates the linker script approach. However, this approach was found to be too buggy.

Autopackage eventually went defunct because of resistance from distributors. I guess that people weren't truly interested in Linux succeeding on the desktop, despite how many people complained about it.

Fast forward to 2015. Virtualization and containerization became cheap and ubiquitous. Thus the holy build box approach became viable to the average developer.

Other comparable systems

We have heard that the Meson build system also allows compiling cross-distribution Linux binaries. Meson is more fully-featured than Holy Build Box in that it tries to abstract away library and dependency management.

One of the most interesting features in Meson is that it automatically generates a dependency manifest when you use it to create statically linked binaries. This manifest allows you to find out which library versions a binary is linked to, which in turn allows you to automatically scan binaries for security vulnerabilities. Holy Build Box may follow this approach some time in the future.

holy-build-box's Issues

Yum update is failing - Centos 6 EOL

Hello! The yum install/update commands stopped working for us recently. I suspect that this may be related to the EOL of the Centos 6 base image but I'm not sure.

The error is:

# yum update
Loaded plugins: fastestmirror, ovl
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

Is there any plan of upgrading the base image? Or maybe setting the vault by default?

Requesting bearer token: invalid status code from registry 403 (Forbidden)

It took a while for me to find out that this link is wrong, and the correct one in #60 is not merged.

The correct link is ghcr.io/phusion/holy-build-box/hbb-64.

Unable to clone https urls

The version of git available in holy-build-box does not support https urls.

$ docker run -it --rm --volume ${PWD}:/data phusion/holy-build-box-64:3.0.2 bash
[root@cd0a66973054 /]# source /hbb_exe/activate
Holy build box activated
Prefix: /hbb_exe
CFLAGS/CXXFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
LDFLAGS: -L/hbb_exe/lib -static-libstdc++
STATICLIB_CFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
SHLIB_CFLAGS: -g -O2 -fvisibility=hidden -I/hbb_exe/include 
SHLIB_LDFLAGS: -L/hbb_exe/lib -static-libstdc++

[root@cd0a66973054 /]# git clone https://github.com/google/googletest.git
Cloning into 'googletest'...
git: 'remote-https' is not a git command. See 'git --help'.

This probably has something to do with the version of curl being available when git is built. I was able to work around it by installing curl-devel:

diff --git a/image/build.sh b/image/build.sh
index d31e6eb..09c37c2 100755
--- a/image/build.sh
+++ b/image/build.sh
@@ -65,7 +65,7 @@ if ! eval_bool "$SKIP_INITIALIZE"; then
        run yum update -y
        run yum install -y tar curl m4 autoconf automake libtool pkgconfig openssl-devel \
                file patch bzip2 zlib-devel gettext python-setuptools python-devel \
-               epel-release centos-release-scl
+               epel-release centos-release-scl curl-devel
        run yum install -y python2-pip "devtoolset-$DEVTOOLSET_VERSION"
 fi

but not sure if that is an acceptable solution since curl is also built locally.

grpc build fails on holy-build-box

The build is failing for the following subject :
"FAILED: bin/Linux_x64_Debug/grpc_ruby_plugin"
with the following error :
15:55:25 : && /opt/rh/devtoolset-7/root/usr/bin/c++ -g -O2 -fvisibility=hidden -I/hbb_exe/include -std=c++14 -Wno-unknown-pragmas -Werror=return-type -std=c++11 -g -D_DEBUG -L/hbb_shlib/lib -static-libstdc++ -rdynamic common/third_party/grpc/CMakeFiles/grpc_ruby_plugin.dir/src/compiler/ruby_plugin.cc.o -o bin/Linux_x64_Debug/grpc_ruby_plugin lib/Linux_x64_Debug/libprotocd.a lib/Linux_x64_Debug/libprotobufd.a -ldl -lrt -lm -lpthread lib/Linux_x64_Debug/libgrpc_plugin_support.a lib/Linux_x64_Debug/libprotocd.a lib/Linux_x64_Debug/libprotobufd.a -lpthread -ldl -lrt -lm -lpthread && :
15:55:25 lib/Linux_x64_Debug/libprotobufd.a(common.cc.o): In function void std::call_once<void (*&)()>(std::once_flag&, void (*&)())': 15:55:25 /opt/rh/devtoolset-7/root/usr/include/c++/7/mutex:676: undefined reference to std::__once_callable'
15:55:25 /opt/rh/devtoolset-7/root/usr/include/c++/7/mutex:677: undefined reference to `std::__once_call'

attaching the ENV output for reference
[root@5f731cfd1ed0 lib]# env
CPLUS_INCLUDE_PATH=/hbb_exe/include
LDFLAGS=-L/hbb_exe/lib -static-libstdc++
MANPATH=/opt/rh/devtoolset-7/root/usr/share/man:
HOSTNAME=5f731cfd1ed0
TERM=xterm
CPPFLAGS=-I/hbb_exe/include
PERL5LIB=/opt/rh/devtoolset-7/root//usr/lib64/perl5/vendor_perl:/opt/rh/devtoolset-7/root/usr/lib/perl5:/opt/rh/devtoolset-7/root//usr/share/perl5/vendor_perl
LIBRARY_PATH=/hbb_exe/lib
LDPATHFLAGS=-L/hbb_exe/lib
STATICLIB_CFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
OLDPWD=/hbb_exe
SHLIB_CXXFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
PCP_DIR=/opt/rh/devtoolset-7/root
LD_LIBRARY_PATH=/hbb/lib:/hbb_exe/lib
O3_ALLOWED=true
CXXFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
PATH=/hbb_exe/bin:/hbb/bin:/opt/rh/devtoolset-7/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin
C_INCLUDE_PATH=/hbb_exe/include
PWD=/hbb_exe/lib
STATICLIB_CXXFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
LANG=en_US.UTF-8
SHLIB_LDFLAGS=-L/hbb_exe/lib -static-libstdc++
HOME=/root
SHLVL=1
SHLIB_CFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
GOROOT=/usr/local/go
CFLAGS=-g -O2 -fvisibility=hidden -I/hbb_exe/include
PYTHONPATH=/opt/rh/devtoolset-7/root/usr/lib64/python2.6/site-packages:/opt/rh/devtoolset-7/root/usr/lib/python2.6/site-packages
LESSOPEN=||/usr/bin/lesspipe.sh %s
PKG_CONFIG_PATH=/hbb_exe/lib/pkgconfig:/usr/lib/pkgconfig
INFOPATH=/opt/rh/devtoolset-7/root/usr/share/info
G_BROKEN_FILENAMES=1
_=/usr/bin/env

---

note that If I delete the " LDFLAGS " the build is successful but then it is using the standard OS libstdc++ file and not the holy build box so it king of misses the point ....

---

arm64 version?

Hello, are there any plans to add an arm64 version of the tool?

Undefined references to nearly all TBB symbols when built under holy build box

Hi,

We use holy build box to build distributable linux executables for our software salmon. Recently, we have been trying to update to a more recent version of the threading building blocks library (TBB), since some parts of the API have been deprecated in newer versions. Unfortunately, after making this update, essentially all TBB-related symbols now fail to resolve when building under holy build box.

I recently posted a full rundown of the issue in the TBB repo here. While I am not sure the ultimate cause of the problem, I can only replicate this issue reliably within holy build box. On our server locally (ubuntu 16.04), on my Mac laptop, and on the GitHub CI images we use (ubuntu latest and OSX latest), everything builds cleanly. Yet, in holy build box, we simply cannot get this library to link. Interestingly, this was not a problem with the older versions of TBB, but those are now deprecated and maintenance & development is only done on the new versions, so we cannot stick with the old library in perpetuity.

Any insight you might have into this would be greatly appreciated.

Thanks!
Rob

Update git version in the image

There were bug fixes recently in git, the version installed in hbb images should be updated.

GitHub Pages Link outdated

The link on the right side should be https://phusion.github.io/holy-build-box/

Drop system OpenSSL and system curl

The system OpenSSL and system curl were only installed because CentOS 5's OpenSSL and curl didn't support SNI, which prevented the build script from downloading source code.

According to https://stackoverflow.com/a/12942331/20816, SNI requires:

• curl >= 7.18.1. CentOS 6 has 7.19.7.
• OpenSSL >= 0.9.8j. CentOS 6 has 1.0.1e.

CI failing

The GitHub actions have been failing for a year now.

I also don't know where to find the old logs of past successful CI runs. If those were available, one might have a better chance to find and fix the issue...

Zlib security issue

As in CVE-2018-25032 decribed there is a security issue in zlib version 1.2.11 and below.
A few days ago a new zlib version 1.2.12 was released that fixes this issue.
Can you please update zlib to version 1.2.12 for the next release.

compiling samtools/htslib

Hello,

This isn't so much an issue as a cry for help.

I'm trying to compile htslib from samtools http://www.htslib.org/download/, and I'm getting an error on htslib. In general I'm hoping to use holy-build-box to build bioinformaitcs software.

if test -n "/io/binary/libexec/htslib"; then mkdir -p -m 755 /io/binary/libexec/htslib; fi
gcc -shared -Wl,-soname,libhts.so.1 -pthread -L/hbb_exe -static-libstdc++ -rdynamic -o libhts.so kfunc.pico knetfile.pico kstring.pico bgzf.pico faidx.pico hfile.pico hfile_net.pico hts.pico md5.pico regidx.pico sam.pico synced_bcf_reader.pico vcf_sweep.pico tbx.pico vcf.pico vcfutils.pico cram/cram_codecs.pico cram/cram_decode.pico cram/cram_encode.pico cram/cram_external.pico cram/cram_index.pico cram/cram_io.pico cram/cram_samtools.pico cram/cram_stats.pico cram/files.pico cram/mFILE.pico cram/open_trace_file.pico cram/pooled_alloc.pico cram/rANS_static.pico cram/sam_header.pico cram/string_alloc.pico cram/thread_pool.pico cram/vlen.pico cram/zfio.pico plugin.pico -lz -lm -ldl
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: /hbb_exe/lib/libz.a(crc32.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/hbb_exe/lib/libz.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

Any insight would be very appreciated!

Best,
Jillian

std::thread support not present in /hbb_exe/lib/libstdc++.a

Not too sure about this one, but I have a C++ application that I'm building with holy-build-box. I'm having an issue when trying to link code that uses std::call_once:
...
/opt/rh/devtoolset-8/root/usr/include/c++/8/mutex:677: undefined reference to `std::__once_call'
...

It appears the /hbb_exe/lib/libstdc++.a may not have thread support, compare:

nm -C /hbb_exe/lib/libstdc++.a | grep std::thread
and
nm -C /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libstdc++.a | grep std::thread
nm: compatibility-debug_list-2.o: no symbols

Not Working With Docker For Windows

Hi,
I am trying to follow the get started guide on my windows 10's Ubuntu WSL 2
And The get started Guide didn't Worked.

But I also tried it on a ubuntu server it worked just fine.
This problem is occurring on Docker on Windows (WSL2)

Old libstdc++ is shipped

Latest holy-build-box upgraded to devtoolset 9, however, still ships libstdc++ 8.3.0. This causes all sorts of link errors are library and the includes (system!) do not match.

Compiling:

#include <string>                                                                                                                                                                                                     #include <iostream>
#include <sstream>

int main(void) {
  std::stringstream s;
  s << "Foo";
  std::cout << "Hello world!\n" << s.str() << std::endl;
  return 0;
}

With g++ test.cpp -static-libstdc++ -static-libgcc -O2 -fvisibility=hidden -I/hbb_exe/include -L/hbb_exe/lib

Yields:

/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /tmp/cciMQUab.o: in function `main':
1.cpp:(.text.startup+0xe): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'

The symbol (ctor of basic_stringstream) moved from being inlined to library.

yum update/install fails with phusion/holy-build-box-32:latest image

Noticed today that yum updates/installs aren't working in the 32-bit HBB image. Perhaps the packagecloud subscription wasn't paid?

Error encountered

bash-4.1# yum update
Loaded plugins: fastestmirror
Setting up Update Process
Determining fastest mirrors
epel/metalink                                                                                    | 4.7 kB     00:00
 * epel: mirror.math.princeton.edu
CentOS-base                                                                                      | 3.7 kB     00:00
CentOS-base/primary_db                                                                           | 4.7 MB     00:03
CentOS-centosplus                                                                                | 3.4 kB     00:00
CentOS-centosplus/primary_db                                                                     | 5.7 MB     00:03
CentOS-contrib                                                                                   | 2.9 kB     00:00
CentOS-contrib/primary_db                                                                        | 7.4 kB     00:00
CentOS-extras                                                                                    | 3.4 kB     00:00
CentOS-extras/primary_db                                                                         |  29 kB     00:00
CentOS-fasttrack                                                                                 | 2.9 kB     00:00
CentOS-fasttrack/primary_db                                                                      | 9.9 kB     00:00
CentOS-updates                                                                                   | 3.4 kB     00:00
CentOS-updates/primary_db                                                                        |  12 MB     00:07
epel                                                                                             | 4.7 kB     00:00
epel/primary_db                                                                                  | 6.1 MB     00:00
https://packagecloud.io/phusion/centos-6-scl-i386/el/6/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 402 Payment Required"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: phusion_centos-6-scl-i386. Please verify its path and try again

Workarounds:

rm -f /etc/yum.repos.d/phusion_centos-6-scl-i386.repo
to remove the repo before issuing any yum commands

Alternatively, issue yum commands with the --disablerepo flag:
yum --disablerepo=phusion_centos-6-scl-i386,phusion_centos-6-scl-i386-source install <package>

Q: Are very recent distributions really supported?

The FAQ states that binaries should be ok on Debian >6, Ubuntu >10, which would be great.

The question is if binaries were really tested on very new distributions, like ARCH, which is probably the most advanced one. And even Ubuntu 16/17.

libstdc++ version doesn't match the one in devtoolset-8

We compile libstdc++ 8.3.0, but it should be 8.3.1 in order to match the version provided by devtoolset-8.

How to use hbb with visual studio

It would be great if holy buildbox could be integrated with visual studio to generate a portable executable. That would require cmake, vcpkg, openssh-server g++ gdb make ninja-build rsync zip.

I have tried modifying the image to install ssh and run visual studio but somehow the ssh connections are not very stable.

new docker image needs published

i'm building from your phusion/holy-build-box-64:latest and it has been amazing. I appreciate the efforts. Recently I changed my requirements for cmake to version 3.21+. I see in the merge log you updated it to 3.22 a year ago but the latest docker image is still at 3.19.

Update github pages link in repo description

to
https://phusion.github.io/holy-build-box/

Add a license

The project currently does not have a license, and as a result I am not sure what limitations there are (if any) for using the software.

Update to a supported OS

As you may be aware CentOS 5 is no longer supported. I was wondering if its possible to update to a supported OS?

Phusion PackageCloud repo will go away / drop x86 / upgrade to CentOS 7

The build script references https://packagecloud.io/install/repositories/phusion/centos-6-scl-i386/script.rpm.sh, but this repository will go away in less than a year.

I think we should either drop x86 support altogether, or find an alternative. I think dropping x86 isn't so crazy nowadays: even years ago, I've observed that nearly nobody uses the x86 binaries for Passenger. I think that going forward, we're more likely to want to support arm64 instead.

Maintainers wanted?

Hi! First off, thanks! This docker container is awesome, basically exactly what we needed to automate building some binaries for releases that work on a wider range of Linux systems (there were some minor issues like a bug in gcc 7.3.1 that were easy to work around).

I noticed the readme has a maintainers wanted badge -- what do you need help with? It looks like having a CentOS 6 container will be needed for the foreseeable future and I'd rather help keep an existing project up to date than create and maintain a fork.

After transferring the binary, the error: segmentation fault

For the test, I use an example from their documentation https://github.com/phusion/holy-build-box/blob/master/TUTORIAL-1-BASICS.md Compilation is performed on the Ubuntu + Docker virtual machine. After compilation, launch on Ubuntu goes fine, everything works. When migrating to another Linux host (CentOS), when starting the binary, I get an error: segmentation fault. What needs to be done to make this binary work normally?

Unable to compile librdkafka

Hi,
I am unable to compile librdkafka (https://github.com/edenhill/librdkafka).
Steps:
I have logged into the container and activated hbb_exe
Configure of librdkafka throws out C++ not found

Cannot start HBB in Arch Linux with kernel 4.20.11

When running the basic command presented in the tutorials:

$ docker run -t -i --rm phusion/holy-build-box-64:latest bash

absolutely nothing happens, the command quits in about ~10 seconds with exit code 139 and docker ps -a displays nothing.

When trying the most simple command:

$ docker run phusion/holy-build-box-64:latest
$ docker ps -a
CONTAINER ID        IMAGE                                          COMMAND                  CREATED             STATUS                        PORTS               NAMES
6657c56132f3        phusion/holy-build-box-64:latest               "/bin/bash"              10 seconds ago      Exited (139) 7 seconds ago                        affectionate_bell

Any idea how to solve this? I'm not a docker expert (;

Consider adding a micro-init system

I think we should add a micro-init system, such as tini, in order to avoid the PID 1 zombie reaping problem. What do you think @nightlark?

activating x86 container on x64 host causes

+ /hbb_exe/activate-exec /hbb/activate_func.sh: line 38: /opt/rh/devtoolset-8/enable: No such file or directory

reason is due to
function activate_holy_build_box_deps_installation_environment() { if [ "$(uname -m)" != x86_64 ]; then DEVTOOLSET_VER=7 else DEVTOOLSET_VER=8 fi
in /hbb/activate_func.sh
devtoolset 8 isn't readily available or in the container for x86, should always be set to 7 for the x86 container.

GCC 4.9

Wow, this project is really something we have been looking for and much simpler than our current collection of outdated centos virtualbox images. It even has up to date cmake 👍

But, we do require at least GCC 4.9, because GCC 4.8 is broken for really many C++11 features. Do you have any plans on updating? I actually think quite a few projects require GCC>=4.9 now...

Thank you very much!

Image parameter in tutorials requires container registry URL

While working through Tutorial 1, I received the following error:

> docker run -t -i --rm foobarwidget/holy-build-box-x64 bash

Unable to find image 'foobarwidget/holy-build-box-x64:latest' locally
docker: Error response from daemon: pull access denied for foobarwidget/holy-build-box-x64, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

I was able to resolve the error by prepending the URL to the Github container repository to the image parameter, like so:

> docker run -t -i --rm ghcr.io/foobarwidget/holy-build-box-x64 bash
Unable to find image 'ghcr.io/foobarwidget/holy-build-box-x64:latest' locally                             
latest: Pulling from foobarwidget/holy-build-box-x64                                                      
2d473b07cdd5: Pull complete                          
35bed6fba959: Pull complete                          
7f8911f81a08: Pull complete                          
Digest: sha256:a51f858721fe5d520f5eaa86979fa61ef7206561164103b8447dfcc2297ad57d                           
Status: Downloaded newer image for ghcr.io/foobarwidget/holy-build-box-x64:latest                         
[root@1337a209ab06 /]#

New users (particularly users new to Docker) might not realize that the container registry URL is needed, and so a documentation update might be in order. I'd be glad to submit a PR but wanted to file an issue first to make sure there wasn't something else I was missing.

Thanks for providing such a nifty utility!