maresb/docker-build-s3fs

Build s3fs from Docker

docker-build-s3fs

Builds a .deb for Ubuntu 18.04/20.04 based on a specified commit from the s3fs-fuse GitHub repository.

This Dockerfile repackages the Ubuntu 20.04 focal .deb file (currently s3fs_1.86-1_amd64.deb) with updated source code from the s3fs-fuse GitHub repository. This is all done at Docker build time.

Download

Download the resulting .deb file (rendered from v1.91 release) and verify your preferred checksum.

Links

• GitHub: https://github.com/maresb/docker-build-s3fs

• s3fs-fuse: https://github.com/s3fs-fuse/s3fs-fuse

Motivation

Ubuntu 18.04's outdated version of s3fs-fuse (resulting from apt-get install s3fs) did not seem to work for me on AWS. Thus I started this project with a Dockerfile which compiles s3fs from a specified GitHub commit.

I need to deploy this to several instances, so I much prefer the speed and convenience of installing from a .deb file as opposed to compiling from source. For security reasons, I avoid downloading software compiled by random people. If you trust me¹, you can download the .deb from the above link. Otherwise, follow the instructions below to build it yourself.

Alternatives

I no longer use this project myself. Nowadays my preferred installation method is to use conda-forge, which provides open-source infrastructure for package management. In particular, I use mamba to install the s3fs-fuse package with the command

mamba install -c conda-forge s3fs-fuse

If you don't want to mess with setting up a Conda environment, the following should work on Ubuntu 22.04 (not thoroughly tested):

sudo apt-get install -y curl bzip2 fuse mime-support libxml2 libcurl4 libssl3
curl -Ls https://anaconda.org/conda-forge/s3fs-fuse/1.91/download/linux-64/s3fs-fuse-1.91-h66f7299_0.tar.bz2 | sudo tar -xvj --strip-components=1 --directory=/usr/local/bin bin/s3fs

On Ubuntu 20.04, libssl3 is not available, but the following procedure should install it from Conda-Forge (not thoroughly tested):

DEBIAN_FRONTEND=noninteractive sudo apt-get install -y curl bzip2 fuse mime-support libxml2 libcurl4
curl -Ls https://anaconda.org/conda-forge/openssl/3.0.3/download/linux-64/openssl-3.0.3-h166bdaf_0.tar.bz2 | sudo tar -xvj --strip-components=1 --directory=/usr/local/lib --wildcards "lib/lib*"
curl -Ls https://anaconda.org/conda-forge/s3fs-fuse/1.91/download/linux-64/s3fs-fuse-1.91-h66f7299_0.tar.bz2 | sudo tar -xvj --strip-components=1 --directory=/usr/local/bin bin/s3fs

Installation

Once you have the .deb file, run

sudo apt-get install -y fuse mime-support libxml2 libcurl4 libssl1.1
sudo dpkg -i s3fs_….deb

Build s3fs from Docker

If you wish to validate my .deb file or to generate it yourself, then follow these steps.

1. Compile under Docker.

Download Dockerfile and change to the corresponding directory.

git clone https://github.com/maresb/docker-build-s3fs.git && cd docker-build-s3fs

To render an untagged commit such as e0712f4,

docker build -t build-s3fs --build-arg COMMIT_ID=e0712f4 --build-arg S3FS_VERSION=1.85 .

or for a tagged commit such as a release version, for example v1.91,

docker build -t build-s3fs --build-arg COMMIT_ID=v1.91 --build-arg S3FS_VERSION=1.91 .

The argument S3FS_VERSION should refer to the latest version number as of the commit specified under COMMIT_ID.

2. Copy the package from the image via a temporary container.

id=$(docker create build-s3fs)
docker cp $id:s3fs-debian-package.tar .
docker rm -v $id
tar xvf s3fs-debian-package.tar && rm s3fs-debian-package.tar

3. Clean up.

docker rmi build-s3fs
docker system prune

For debugging,

If the image successfully builds, you can tag the build stage and look inside with

docker build -t build-s3fs:build --target build [ADD BUILD ARGS HERE] .
docker run --rm -it build-s3fs:build /bin/bash

Otherwise, in the output of a partial build, look for a line with an arrow directly followed by a hash such as

 ---> df7f92f1a162

Then you can look inside at the corresponding point with

docker run --rm -it df7f92f1a162 /bin/bash

Checksums

Normally, due to last-modified times of files and the timestamp in the changelog, no two .deb files are expected to be exactly the same, even if they have the exact same content. However, by fixing a build-time, the .deb files produced by this container are reproducible. Thus I can provide a checksum which can be verified from the download, and/or a build on your own computer.

NOTE: After some time, discrepancies in the checksums could arise due to updated dependencies. To compare, I provide logs under builds/.

md5sums FILE CONTENTS

0f511a762f22c0c00d75e40515dca142  usr/bin/s3fs
e225a2dfdaadb1c4c484614df28cdfdf  usr/share/doc/s3fs/changelog.Debian.gz
2c85defc6568bd463d3dc9c5342faab0  usr/share/doc/s3fs/copyright
7ba8239dcc20cbbbe29a0fcb80cc27ed  usr/share/doc/s3fs/examples/passwd-s3fs
0b395c729d822fdfd60bcf0d13169c49  usr/share/man/man1/s3fs.1.gz

s3fs_1.91+git-v1.91-3_amd64.deb size and checksum

$ stat --printf="%s bytes\n" s3fs_1.91+git-v1.91-3_amd64.deb
222340 bytes

$ md5sum s3fs_1.91+git-v1.91-3_amd64.deb
95192b8c47c931623eb1ce49282338eb  s3fs_1.91+git-v1.91-3_amd64.deb

$ sha256sum s3fs_1.91+git-v1.91-3_amd64.deb
9eca7cb1e9bb08b8e3ae9b476287712a8293493630c76d9102c5ac0058b098c0  s3fs_1.91+git-v1.91-3_amd64.deb

$ b2sum s3fs_1.91+git-v1.91-3_amd64.deb
46befabcd4ffce0000db99d71b1d51d6cef2bab8816fc540bc2e7aaaed921ad545f61d661190b96d219648f69688a7ea65ce6142a6524ec3af5a6f7f9f2ea97f  s3fs_1.91+git-v1.91-3_amd64.deb

s3fs size and checksum

$ stat --printf="%s bytes\n" s3fs
604608 bytes

$ md5sum s3fs
0f511a762f22c0c00d75e40515dca142  s3fs

$ sha256sum s3fs
9f205d172c8e23dcf1fa1f31d999ad469116d032ef5238e29a0d290edd91ec07  s3fs

$ b2sum s3fs
c9df12b25a3d577f958b5bb1fd15c406c42344b22fd1e172fb2f1672eca1c0b10b0a3487099f9951ebfaf55a3c81e64ce8a9d2cad1275612d15c5ac1713bae92  s3fs

Originally these files were reproducible by Docker Hub. Unfortunately Docker Hub Autobuilds have been deactivated, so I can no longer offer this service. I would be interested in autogenerating the checksums/binary through another CI service if someone else puts together a pull request.

Notes

[1] On principle, you should check my Dockerfile and make sure that I'm not doing anything suspicious.

docker-build-s3fs's Issues

Multistage build to allow Docker Hub download of just .deb

Right now, the image on Docker Hub consists of a Ubuntu + build tools + prereqs + build products. By adding a build stage at the end, I think I should be able to do something like have an image with just Alpine + .deb file. (Or maybe just .deb file?)

--version commit listed as unknown

Figure out where it comes from and how to feed in the commit...

todo: add BUILD_TIMESTAMP to revisions_to_build.txt

Right now we just build based on a default BUILD_TIMESTAMP. But as soon as that changes, our builds are going to change. We should add it as a pair of columns in revisions_to_build.txt so that we can update BUILD_TIMESTAMP as desired.

Update to 1.90?

A new stable version has been released: https://github.com/s3fs-fuse/s3fs-fuse/releases/tag/v1.90

Add 1.88 build to repo

v1.88 has been released on github.

Can you add this version to the build list and add the .deb package?

I was looking at revisions_to_build.txt and didn't see a line 1.87 even though there's a package uploaded, so I wasn't sure if the file is updated or not. If the version should go in the file, I can make a PR with the new release.

docker-build-s3fs/builds/revisions_to_build.txt

Lines 1 to 6 in 8f7220d

	# This file contains a list of revisions to be built by the do_build.sh script.
	# Each line consists of a Git commit id followed by the corresponding last s3fs
	# release number.
	e0712f4 1.85
	v1.86 1.86

arm64 arch support

Hello,

I'm asking for arm64 architecture support. Is this possible ?

1.91 build

Hello there

Any chance to have the v1.91 build ?

Cheers

Use updated debian packaging?

Currently I'm using the old Ubuntu package as the base. Maybe I should be using this instead:

NOTICE: 's3fs-fuse' packaging is maintained in the 'Git' version control system at:
https://anonscm.debian.org/git/collab-maint/s3fs-fuse.git -b debian
Please use:
git clone https://anonscm.debian.org/git/collab-maint/s3fs-fuse.git -b debian
to retrieve the latest (possibly unreleased) updates to the package.