ocurrent/obuilder-fs

The filesystem for making macOS work with OBuilder

obuilder-fs

obuilder-fs is no longer used to provide obuilder on macos. This project is archived for reference purposes only.

The File System for OBuilder x MacOS -- it's sole purpose is to redirect calls heading to some path (e.g. /usr/local) to a directory pointed to by a symbolic link inside another folder (e.g. /data/scoreboard/) using the UID of the Fuse Context of the caller (e.g. /data/scoreboard/<uid> ~~> /Volumes/tank/result/<hash>). To make this clearer here is a series of steps this FS should take:

(1) Mount obuilder-fs to /usr/local with scoreboard /data/scoreboard 

(2) User mac705 makes a call like: ls /usr/local

(3) obuilder-fs intercepts and calls: readlink on /data/scoreboard/705 

(4) successful readlink returns: /Volumes/tank/result/abcde12345

(5) obuilder-fs list the directories inside /Volumes/tank/result/abcde12345

The CLI tool is very simple as expects <scoreboard-path> <mount-point> <fuse-args-like-allow-other>....

A typical usage would be:

sudo obuilderfs ~/scoreboard /usr/local -o allow_other

obuilder-fs's Issues

Weird installation problems on Big-Sur

Problems when installing basic packages like pkg-config

Full Log

2022-01-07 13:28.12: New job: build reddit_api_kernel.0.2.1, using opam 2.1
                              from https://github.com/ocaml/opam-repository.git#refs/pull/20392/head (6d1dca9527b4ce918c1c21cf984b9176f5967fbb)
                              on macos-homebrew-ocaml-4.13/amd64
To reproduce locally:
git clone --recursive "https://github.com/ocaml/opam-repository.git" && cd "opam-repository" && git fetch origin "refs/pull/20392/head" && git reset --hard 6d1dca95

git fetch origin master

git merge 0c8a79fcbd2cba5ae5fe226bfedcd87d53bdd187

cat > ../Dockerfile <<'END-OF-DOCKERFILE'

FROM macos-homebrew-ocaml-4.13

USER 1000:1000

RUN opam repository remove -a multicore || true

RUN ln -f ~/local/bin/opam-2.1 ~/local/bin/opam

RUN opam init --reinit -ni

ENV OPAMDOWNLOADJOBS 1

ENV OPAMERRLOGLEN 0

ENV OPAMSOLVERTIMEOUT 500

ENV OPAMPRECISETRACKING 1

RUN rm -rf opam-repository/

COPY --chown=1000:1000 . opam-repository/

RUN opam repository set-url --strict default "file://$HOME/opam-repository"

RUN opam pin add -k version -yn reddit_api_kernel.0.2.1 0.2.1

RUN opam update --depexts

RUN opam remove reddit_api_kernel.0.2.1 && opam install --deps-only reddit_api_kernel.0.2.1 && opam install -v reddit_api_kernel.0.2.1; 

res=$?; 

test "$res" != 31 && exit "$res"; 

export OPAMCLI=2.0; 

build_dir=$(opam var prefix)/.opam-switch/build; 

failed=$(ls "$build_dir"); 

for pkg in $failed; do 

if opam show -f x-ci-accept-failures: "$pkg" | grep -qF ""macos-homebrew""; then 

echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field."; 

fi; 

done; 

exit 1
END-OF-DOCKERFILE

docker build -f ../Dockerfile .
2022-01-07 13:28.12: Using cache hint "macos-homebrew-ocaml-4.13-reddit_api_kernel.0.2.1-6d1dca9527b4ce918c1c21cf984b9176f5967fbb"

2022-01-07 13:28.12: Using OBuilder spec:

((from macos-homebrew-ocaml-4.13)

(user (uid 1000) (gid 1000))

(run (shell "opam repository remove -a multicore || true"))

(run (shell "ln -f ~/local/bin/opam-2.1 ~/local/bin/opam"))

(run (shell "opam init --reinit -ni"))

(env OPAMDOWNLOADJOBS 1)

(env OPAMERRLOGLEN 0)

(env OPAMSOLVERTIMEOUT 500)

(env OPAMPRECISETRACKING 1)

(run (shell "rm -rf opam-repository/"))

(copy (src .) (dst opam-repository/))

(run (shell "opam repository set-url --strict default "file://$HOME/opam-repository""))

(run (shell "opam pin add -k version -yn reddit_api_kernel.0.2.1 0.2.1"))

(run (network host)

(shell "opam update --depexts"))

(run (cache (opam-archives (target /home/opam/.opam/download-cache)))

(network host)

(shell  "opam remove reddit_api_kernel.0.2.1 && opam install --deps-only reddit_api_kernel.0.2.1 && opam install -v reddit_api_kernel.0.2.1;

\n        res=$?;

\n        test "$res" != 31 && exit "$res";

\n        export OPAMCLI=2.0;

\n        build_dir=$(opam var prefix)/.opam-switch/build;

\n        failed=$(ls "$build_dir");

\n        for pkg in $failed; do

\n          if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"macos-homebrew\""; then

\n            echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field.";

\n          fi;

\n        done;

\n        exit 1"))

)
2022-01-07 13:28.12: Waiting for resource in pool OCluster

2022-01-07 13:28.13: Waiting for worker...

2022-01-07 13:39.24: Got resource from pool OCluster

Building on mac-sur-2

All commits already cached

Updating files:  21% (5117/23864)

Updating files:  22% (5251/23864)

Updating files:  23% (5489/23864)

Updating files:  24% (5728/23864)

Updating files:  25% (5966/23864)

Updating files:  26% (6205/23864)

Updating files:  27% (6444/23864)

Updating files:  28% (6682/23864)

Updating files:  29% (6921/23864)

Updating files:  30% (7160/23864)

Updating files:  31% (7398/23864)

Updating files:  32% (7637/23864)

Updating files:  33% (7876/23864)

Updating files:  34% (8114/23864)

Updating files:  35% (8353/23864)

Updating files:  36% (8592/23864)

Updating files:  37% (8830/23864)

Updating files:  38% (9069/23864)

Updating files:  39% (9307/23864)

Updating files:  40% (9546/23864)

Updating files:  41% (9785/23864)

Updating files:  42% (10023/23864)

Updating files:  43% (10262/23864)

Updating files:  43% (10395/23864)

Updating files:  44% (10501/23864)

Updating files:  45% (10739/23864)

Updating files:  46% (10978/23864)

Updating files:  47% (11217/23864)

Updating files:  48% (11455/23864)

Updating files:  49% (11694/23864)

Updating files:  50% (11932/23864)

Updating files:  51% (12171/23864)

Updating files:  52% (12410/23864)

Updating files:  53% (12648/23864)

Updating files:  54% (12887/23864)

Updating files:  55% (13126/23864)

Updating files:  56% (13364/23864)

Updating files:  57% (13603/23864)

Updating files:  58% (13842/23864)

Updating files:  59% (14080/23864)

Updating files:  60% (14319/23864)

Updating files:  61% (14558/23864)

Updating files:  62% (14796/23864)

Updating files:  63% (15035/23864)

Updating files:  64% (15273/23864)

Updating files:  65% (15512/23864)

Updating files:  66% (15751/23864)

Updating files:  67% (15989/23864)

Updating files:  68% (16228/23864)

Updating files:  69% (16467/23864)

Updating files:  69% (16592/23864)

Updating files:  70% (16705/23864)

Updating files:  71% (16944/23864)

Updating files:  72% (17183/23864)

Updating files:  73% (17421/23864)

Updating files:  74% (17660/23864)

Updating files:  75% (17898/23864)

Updating files:  76% (18137/23864)

Updating files:  77% (18376/23864)

Updating files:  78% (18614/23864)

Updating files:  79% (18853/23864)

Updating files:  80% (19092/23864)

Updating files:  81% (19330/23864)

Updating files:  82% (19569/23864)

Updating files:  83% (19808/23864)

Updating files:  84% (20046/23864)

Updating files:  85% (20285/23864)

Updating files:  86% (20524/23864)

Updating files:  87% (20762/23864)

Updating files:  88% (21001/23864)

Updating files:  89% (21239/23864)

Updating files:  90% (21478/23864)

Updating files:  91% (21717/23864)

Updating files:  92% (21955/23864)

Updating files:  93% (22194/23864)

Updating files:  94% (22433/23864)

Updating files:  94% (22589/23864)

Updating files:  95% (22671/23864)

Updating files:  96% (22910/23864)

Updating files:  97% (23149/23864)

Updating files:  98% (23387/23864)

Updating files:  99% (23626/23864)

Updating files: 100% (23864/23864)

Updating files: 100% (23864/23864), done.

HEAD is now at 0c8a79fcbd Merge pull request #20398 from hannesm/release-ca-certs-nss-v3.74

Merge made by the 'recursive' strategy.

.../reddit_api_async/reddit_api_async.0.2.1/opam   | 43 ++++++++++++++++++++++

.../reddit_api_kernel/reddit_api_kernel.0.2.1/opam | 37 +++++++++++++++++++

2 files changed, 80 insertions(+)

create mode 100644 packages/reddit_api_async/reddit_api_async.0.2.1/opam

create mode 100644 packages/reddit_api_kernel/reddit_api_kernel.0.2.1/opam
(from macos-homebrew-ocaml-4.13)

2022-01-07 13:39.29 ---> using "d3db4069272450449a3bab0987ac8ece1435f33c4ab20f28b681c8198f00b6e3" from cache
/: (user (uid 1000) (gid 1000))
/: (run (shell "opam repository remove -a multicore || true"))

[WARNING] No configured repositories by these names found: multicore

2022-01-07 13:39.29 ---> using "62ae3499a03a587b1ae0d36df989b3071f1598f47eaa5de191bd5497b75dd3d5" from cache
/: (run (shell "ln -f ~/local/bin/opam-2.1 ~/local/bin/opam"))

2022-01-07 13:39.29 ---> using "3fee1055c28376822e1489d25f7cf1ade73223b047c59339baa7dd024f0e8d0d" from cache
/: (run (shell "opam init --reinit -ni"))

No configuration file found, using built-in defaults.

Checking for available remotes: rsync and local, git.

you won't be able to use mercurial repositories unless you install the hg command on your system.
you won't be able to use darcs repositories unless you install the darcs command on your system.

This version of opam requires an update to the layout of /Users/mac705/.opam from version 2.0 to version 2.1, which can't be reverted.

You may want to back it up before going further.
Continue? [Y/n] y

Format upgrade done.
<><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>

[default] no changes from git+file:///Users/macos-homebrew-ocaml-4.13/opam-repository

2022-01-07 13:39.29 ---> using "de59b9735fcced5a5965f9f96dc419a25c075572e771a99edd1775fa813c347a" from cache
/: (env OPAMDOWNLOADJOBS 1)
/: (env OPAMERRLOGLEN 0)
/: (env OPAMSOLVERTIMEOUT 500)
/: (env OPAMPRECISETRACKING 1)
/: (run (shell "rm -rf opam-repository/"))

2022-01-07 13:39.29 ---> using "665063bb18576c4e91a3eecdaa59c4887f10bff674a2dd63484849cbbefe4ab3" from cache
/: (copy (src .) (dst opam-repository/))

2022-01-07 13:40.44 ---> saved as "e2bcb34f4da760808aafdfb760780c27022909a97dcf4bb5486b89b1b4cf2d9e"
/: (run (shell "opam repository set-url --strict default "file://$HOME/opam-repository""))

[default] Initialised

2022-01-07 13:42.13 ---> saved as "8c4865723d7b61eb39d262b1c6b60cb04e9a65b2ef78fb525f8b18a4d606c923"
/: (run (shell "opam pin add -k version -yn reddit_api_kernel.0.2.1 0.2.1"))

reddit_api_kernel is now pinned to version 0.2.1

2022-01-07 13:43.05 ---> saved as "35cf6a79cfbad3607cd4c9372ab1704a1190f3ed9027da07eb8adfa548e83427"
/: (run (network host)

(shell "opam update --depexts"))

/usr/local/bin/brew "update"


==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
https://github.com/Homebrew/brew#donations
Updated 1 tap (homebrew/core).
==> New Formulae
arkade
bash-language-server
ctlptl
dtm
fnt
fprettify
fypp
grafana-agent
ivtools
libgrape-lite
mold
nerdctl
rune
yaml-language-server
zbctl
zsh-fast-syntax-highlighting
==> Updated Formulae
alda
all-repos
alluxio
[email protected]
[email protected]
apktool
argocd
argocd-autopilot
argyll-cms
armadillo
asciidoctor
ask-cli
assh
assimp
asymptote
atlantis
avrdude
aws-cdk
aws-elasticbeanstalk
awscli
azure-cli
babel
baidupcs-go
bash
bash_unit
bear
bee
benthos
bgpq3
bigloo
borgmatic
bosh-cli
brigade-cli
broot
btop
byacc
cargo-bloat
cargo-llvm-lines
catch2
cbmc
ccextractor
cdk8s
cdktf
cflow
cgif
checkov
chezmoi
clarinet
clash
clojure
clojurescript
closure-compiler
cloud-nuke
cloudflare-wrangler
code-minimap
composer
conan
conftest
cppad
crane
crystal
cspice
css-crush
cubejs-cli
cyrus-sasl
czmq
dar
datree
dbhash
deno
dependency-check
diffoscope
direnv
django-completion
docker-compose
dolt
dosbox-x
doxygen
drone-cli
dua-cli
duplicity
dwarfutils
easy-tag
efm-langserver
elasticsearch@6
esbuild
eslint
esptool
ethereum
exiv2
exploitdb
f3d
fabric
faudio
fd
fdroidserver
feh
ffmpeg
fisher
fits
fizz
flatbuffers
flintrock
flow
flyway
folly
freetds
gallery-dl
gauche
gdal
gifski
gimme-aws-creds
ginac
git-annex
git-revise
gitbackup
gitg
gnupg-pkcs11-scd
[email protected]
goawk
gobject-introspection
goplus
goreleaser
gphoto2
gradle-profiler
graph-tool
gspell
guile
h2
halide
hidapi
hlint
htmlq
htpdate
igraph
inform6
infracost
inspircd
ioctl
ipmiutil
ipython
iso-codes
jc
jenkins
jfrog-cli
jhipster
jo
jrnl
jsonschema
k3sup
kalker
kamel
ki
kitchen-sync
knot-resolver
kondo
lablgtk
languagetool
latex2html
leaf-proxy
leakcanary-shark
lesspipe
libdap
libgit2-glib
libgphoto2
libgr
libid3tag
libmp3splt
libngspice
libpano
libpipeline
libplctag
librist
libserdes
libsoup
libunistring
libuv
libvirt
liquidctl
lmod
localstack
luau
macchina
madplay
magic-wormhole
mailutils
mame
marked
mat2
maturin
mavsdk
mbedtls
mdbook
mdzk
mercurial
mesa
micronaut
minidlna
minio
minio-mc
mlkit
mmark
moco
molecule
mongo-c-driver
mono
mp3fs
mpd
mpg321
mpv
mtr
mutt
mycli
nanoflann
nativefier
ncdu
ncspot
nebula
neovim
netlify-cli
newrelic-cli
nexus
nginx
ngspice
nlohmann-json
node-sass
node@16
notcurses
nss
numpy
nvm
obfs4proxy
oci-cli
oha
oil
opa
opencv
openstackclient
oq
orientdb
ortp
overdrive
pacapt
passenger
pdm
percona-server
pgpool-ii
phoronix-test-suite
phpstan
picard-tools
pillow
pipx
pkcs11-helper
pnpm
po4a
ponyc
powerman
prestd
proj
protobuf
psalm
pueue
pulumi
pwntools
pygments
pyright
pyside
rabbitmq
rav1e
rebar3
remind
reminiscence
richmd
rizin
rke
rollup
rom-tools
root
rqlite
rush
rust
rust-analyzer
s-search
sail
sbt
scotch
scummvm
sdl2_mixer
seaweedfs
simutrans
snort
socat
sonarqube
spack
spotbugs
sqldiff
sqlite
sqlite-analyzer
sqlmap
sqlx-cli
stormssh
stress-ng
structurizr-cli
stylua
supertux
supervisor
swagger-codegen
swagger-codegen@2
systemd
t-rec
taskwarrior-tui
tbb
tcl-tk
tcsh
tdlib
tealdeer
teku
telegraf
termius
terraform
terraform-ls
terrascan
tesseract
thefuck
theharvester
tile38
tkdiff
tomee-plus
tomee-webprofile
translate-toolkit
travis
tree
tree-sitter
treefrog
twarc
txr
ugrep
universal-ctags
uriparser
usb.ids
vala-language-server
vapoursynth-ocr
vim
virtualenv
vite
vsts-cli
wakatime-cli
wakeonlan
wallpaper
wangle
wasmtime
watchman
webtorrent-cli
whistle
whois
wireshark
wolfssl
wtfutil
yaze-ag
zellij
==> Renamed Formulae
mr2 -> zoro
==> Deleted Formulae
appledoc
rawtoaces

2022-01-07 13:44.46 ---> saved as "31eb831bf80bac922d5d43a783a52008db88d6d644e543c939ce57fcd56c8c28"

/: (run (cache (opam-archives (target /home/opam/.opam/download-cache)))

(network host)

(shell  "opam remove reddit_api_kernel.0.2.1 && opam install --deps-only reddit_api_kernel.0.2.1 && opam install -v reddit_api_kernel.0.2.1;

\n        res=$?;

\n        test "$res" != 31 && exit "$res";

\n        export OPAMCLI=2.0;

\n        build_dir=$(opam var prefix)/.opam-switch/build;

\n        failed=$(ls "$build_dir");

\n        for pkg in $failed; do

\n          if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"macos-homebrew\""; then

\n            echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field.";

\n          fi;

\n        done;

\n        exit 1"))

[NOTE] reddit_api_kernel.0.2.1 is not installed.
Nothing to do.

The following actions will be performed:

install seq                 base
install ocamlbuild          0.14.0
install conf-pkg-config     2
install ocamlfind           1.9.1
install dune                2.9.1
install uchar               0.0.2
install topkg               1.0.4
install num                 1.4
install base-bytes          base
install stdlib-shims        0.3.0
install sexplib0            v0.14.0
install result              1.5
install re                  1.10.3
install ppx_derivers        1.2.1
install octavius            1.2.2
install ocaml-syntax-shims  1.0.0
install ocaml-compiler-libs v0.12.4
install jane-street-headers v0.14.0
install csexp               1.5.1
install bigarray-compat     1.0.0
install base64              3.5.0
install uutf                1.0.2
install stringext           1.6.0
install ppxlib              0.24.0
install dune-configurator   2.9.1
install cstruct             6.0.1
install bigstringaf         0.8.0
install jsonm               1.0.1
install ppx_pipebang        v0.14.0
install base                v0.14.2
install hex                 1.4.0
install angstrom            0.15.0
install variantslib         v0.14.0
install typerep             v0.14.0
install stdio               v0.14.0
install ppx_stable          v0.14.1
install ppx_sexp_conv       v0.14.3
install ppx_optional        v0.14.0
install ppx_let             v0.14.0
install ppx_js_style        v0.14.1
install ppx_here            v0.14.0
install ppx_fixed_literal   v0.14.0
install ppx_enumerate       v0.14.0
install ppx_compare         v0.14.0
install ppx_cold            v0.14.0
install parsexp             v0.14.1
install fieldslib           v0.14.0
install ezjsonm             1.3.0
install uri                 4.2.0
install ppx_variants_conv   v0.14.2
install ppx_typerep_conv    v0.14.2
install ppx_optcomp         v0.14.3
install ppx_custom_printf   v0.14.1
install ppx_sexp_value      v0.14.0
install ppx_sexp_message    v0.14.1
install ppx_hash            v0.14.0
install ppx_assert          v0.14.0
install sexplib             v0.14.0
install ppx_fields_conv     v0.14.2
install uri-sexp            4.2.0
install ppx_base            v0.14.0
install jst-config          v0.14.1
install bin_prot            v0.14.0
install cohttp              5.0.0
install ppx_string          v0.14.1
install time_now            v0.14.0
install ppx_bin_prot        v0.14.0
install ppx_module_timer    v0.14.0
install ppx_inline_test     v0.14.1
install ppx_expect          v0.14.2
install ppx_bench           v0.14.1
install splittable_random   v0.14.0
install base_quickcheck     v0.14.1
install ppx_jane            v0.14.0
install base_bigstring      v0.14.0
install core_kernel         v0.14.2

===== 76 to install =====

The following system packages will first need to be installed:

pkg-config
<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>

Let opam run your package manager to install the required system packages?

(answer 'n' for other options) [Y/n] y

/usr/local/bin/brew "install" "pkg-config"


==> Downloading https://ghcr.io/v2/homebrew/core/pkg-config/manifests/0.29.2_3
==> Downloading https://ghcr.io/v2/homebrew/core/pkg-config/blobs/sha256:0040b6ebe07f60549800b211343fd5fb3cf83c866d9f62e40f5fb2f38b71e161
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:0040b6ebe07f60549800b211343fd5fb3cf83c866d9f62e40f5fb2f38b71e161?se=2022-01-07T13%3A55%3A00Z&sig=7G8u4Lj9xTK8PiZlrCm9HajqOYK5bN6tpnlf%2BvOf0uQ%3D&sp=r&spr=https&sr=b&sv=2019-12-12
==> Pouring pkg-config--0.29.2_3.big_sur.bottle.tar.gz
Error: Failed to fix install linkage
The formula built, but you may encounter issues using it or linking other
formulae against it.
==> Summary
🍺  /usr/local/Cellar/pkg-config/0.29.2_3: 11 files, 656.8KB
==> Running brew cleanup pkg-config...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).

[ERROR] System package install failed with exit code 1 at command:

brew install pkg-config

You can retry with '--assume-depexts' to skip this check, or run 'opam option depext=false' to permanently disable handling of system packages altogether.

"/bin/bash" "-c"

"opam remove reddit_api_kernel.0.2.1 && opam install --deps-only reddit_api_kernel.0.2.1 && opam install -v reddit_api_kernel.0.2.1;

res=$?;

test "$res" != 31 && exit "$res";

export OPAMCLI=2.0;

build_dir=$(opam var prefix)/.opam-switch/build;

failed=$(ls "$build_dir");

for pkg in $failed; do

if opam show -f x-ci-accept-failures: "$pkg" | grep -qF ""macos-homebrew""; then

echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field.";

fi;

done;

exit 1" failed with exit status 10

2022-01-07 13:46.18: Job failed: Failed: Build failed

2022-01-07 13:46.18: Log analysis:

2022-01-07 13:46.18: >>> Error: Failed to fix install linkage (score = 10)

2022-01-07 13:46.18: >>> [ERROR] System package install failed with exit code 1 at command: (score = 10)

2022-01-07 13:46.18: Failed to fix install linkage