mic92/nix-build-uncached

A CI friendly wrapper around nix-build.

nix-build-uncached

Deprecation nix-eval-jobs exposes the binary cache information now also in --check-cache-status, which allows to build a similar feature. nix-fast-build uses nix-eval-jobs and exposes the --skip-cached flag to replace nix-build-uncached. Otherwise NixOS/nix#3946 might track information about possible features in nix itself.

nix-build by default will download already built packages, resulting in unnecessary downloads even if no package has been changed. nix-build-uncached will only build packages not yet in binary caches.

USAGE

nix-build-uncached is available in nixpkgs.

For nix versions bigger than 2.3.x you need to enable at least experimental-features = nix-command in your /etc/nix/nix.conf or $HOME/.config/nix/nix.conf or pass -build-flags "--experimental-features nix-command" to nix-build-uncached.

In the following example ci.nix contains all expressions that should be built. Since only hello-nur is not yet in the binary cache, all other packages are skipped.

[joerg@turingmachine] nix-build-uncached ci.nix
$ nix-build --dry-run ci.nix --keep-going
these derivations will be built:
  /nix/store/s5alllpjx9fmdj26mf9cmxzs3xyxjn7f-hello-2.00.tar.gz.drv
  /nix/store/03m2lwg4zia58zqm7hqlb3r0cgfq53cn-hello-2.00.drv
these paths will be fetched (198.28 MiB download, 681.37 MiB unpacked):
  /nix/store/0mijq2b50xmgk6akxdrg3x8x0k7784jb-python3.8-kiwisolver-1.2.0
  /nix/store/0q1g21q160w2cj2745r24pfn2yb8pmda-python3.8-jupyter_client-6.1.5
  /nix/store/0qxi1gzv1xgpxy58nfk9pxlfqybv1198-cntr-1.2.0
 # ...
$ nix build --keep-going /nix/store/s5alllpjx9fmdj26mf9cmxzs3xyxjn7f-hello-2.00.tar.gz.drv /nix/store/03m2lwg4zia58zqm7hqlb3r0cgfq53cn-hello-2.00.drv
[1/2 built, 1 copied (0.7 MiB)] connecting to 'ssh://[email protected]'

We can pass all arguments that also nix-build accept:

[joerg@turingmachine] nix-build-uncached -E 'with import <nixpkgs> {}; hello'
$ nix-build --dry-run -E with import <nixpkgs> {}; hello --keep-going
these paths will be fetched (0.04 MiB download, 0.20 MiB unpacked):
  /nix/store/aldyr0pjzqydf1vn9lzz7p5gvc141fhn-hello-2.10

However this only affects the evaluation during the dry build, if you want to pass arguments to the final nix build instead, use -build-flags:

[joerg@turingmachine] nix-build-uncached -build-flags '--builders ""' ci.nix
$ nix-build --dry-run ci.nix --builders
these derivations will be built:
  /nix/store/s5alllpjx9fmdj26mf9cmxzs3xyxjn7f-hello-2.00.tar.gz.drv
  /nix/store/03m2lwg4zia58zqm7hqlb3r0cgfq53cn-hello-2.00.drv
these paths will be fetched (198.28 MiB download, 681.37 MiB unpacked):
  /nix/store/0mijq2b50xmgk6akxdrg3x8x0k7784jb-python3.8-kiwisolver-1.2.0
  /nix/store/0q1g21q160w2cj2745r24pfn2yb8pmda-python3.8-jupyter_client-6.1.5
  /nix/store/0qxi1gzv1xgpxy58nfk9pxlfqybv1198-cntr-1.2.0
 # ...
$ nix build --builders   /nix/store/s5alllpjx9fmdj26mf9cmxzs3xyxjn7f-hello-2.00.tar.gz.drv /nix/store/03m2lwg4zia58zqm7hqlb3r0cgfq53cn-hello-2.00.drv
[1/2 built, 1 copied (0.7 MiB)] connecting to 'ssh://[email protected]'

Flakes

We cannot support flakes directly at the time because nix-build does not accept those and nix build's --no-dry-run is broken. However it is possible to add a wrapper nix expression that imports a flake. The following example imports hydra jobs from a nix flake the same directory. It assumes that the flake also has nixpkgs in its inputs.

let
  outputs = builtins.getFlake (toString ./.);
  pkgs = outputs.inputs.nixpkgs;
  drvs = pkgs.lib.collect pkgs.lib.isDerivation outputs.hydraJobs;
in drvs

$ nix-build-uncached ./ci.nix

Packages with allowSubstitutes = false

If your package set you are building has packages at the top level scope that have the attribute allowSubstitutes = false; set, than nix-build-uncached will build/download them everytime. This attribute is set for some builders such as writeText or writeScriptBin. A workaround is to use the following nix library and save it as force_cached.nix. Than wrap your attribute set like this:

let
  pkgs = import <nixpkgs> {};
in (pkgs.callPackage ./force_cached.nix {}) {
  hello = pkgs.writeScriptBin "hello" ''
    #!/bin/sh
    exec ${pkgs.hello}/bin/hello
  '';
}

Real-world examples

• Using nix-build-uncached in github actions

nix-build-uncached's Issues

flakes: weird behavior with github actions nix-build-uncached

With the github action, nix develop -c nix-build-uncached ci.nix doesn't seem to build much. I cleared my cachix cache.

I added a step to show the output of nix-build ci.nix --dry-run. It's supposed to show the same thing that would be built, right?

https://github.com/bbigras/nix-config/runs/1466357334?check_suite_focus=true

Any ideas? Maybe everything I use is in caches.

I wonder if my ci.nix file is correct. I had to change the hydra thing to:

let
  outputs = builtins.getFlake (toString ./.);
  pkgs = outputs.inputs.nixpkgs;
  drvs = pkgs.lib.collect pkgs.lib.isDerivation outputs.deploy;
in
drvs

How to use with niv (without <nixpkgs>)?

Any idea how to use "nix run" with niv?

I'm currently using the following but it would be nice to be able to use it with "skip_adding_nixpkgs_channel": true:

https://github.com/bbigras/nix-config/blob/91239cc64a25cc695e395d96d7947894c2ed0327/.github/workflows/ci.yml#L35-L38

Investigate complete flakes support via `allow-import-from-derivation` option

Just wanted to file an issue to make sure this doesn't get forgotten (since I would really love to use nix-build-uncached with flakes).

From https://logs.nix.samueldr.com/nixos/2020-09-02#1599008399-1599008432 and https://logs.nix.samueldr.com/nixos/2020-09-02#1599027174-1599027205:

00:59 <cole-h> Mic92: re: https://github.com/NixOS/nix/issues/3946#issuecomment-677598130, there is a flag `--allow-import-from-derivation` that can be used
01:00 <cole-h> (I was just trying to use `nix-build-uncached` with flakes and came across the reason why `nix-build` is used, so maybe that can help)
...
06:12 <Mic92> cole-h: did not now that.
06:13 <Mic92> Then I should switch nix-build-uncached to that.
06:13 <cole-h> Mic92: Might make nix-build-uncached with flakes (without a shim) work :D
06:13 <Mic92> it will

Special return on no-rebuild

I have a use-case where I want to only run a command if nix-build-uncached has rebuilt something.

Basically:

• rebuild=$(nix-build-uncached)
• if $rebuild; then cabal build; fi

Potential implementations:

• reserve a special exit status that is returned on no-rebuild, IFF a special flag is also passed to nix-build-uncached. Eg: --status. Does this have the risk of having collisions with other types of build errors?
• write to a file. nix-build-uncache --status ./file

Cannot import path: invalid

At https://github.com/NorfairKing/smos/runs/1044625796?check_suite_focus=true
I'm getting this error:

 error: cannot import '/nix/store/ma3b77vjls3qrznwh66ix5ivvybx6003-source', since path '/nix/store/sqfs8j0g7bmkir2jf2l64rvm3xdfng52-source.drv' is not valid, at /nix/store/pswlp0b96blwb7ab07ihdr7ridf3a00l-source/nix/default.nix:26:1

While the regular nix build passes.

Cannot build on nix 2.3.4

Hey! I tried to install this locally in my project and couldn't get this working with the following derivation:

{ pkgs ? import ./pkgs.nix }:

pkgs.stdenv.mkDerivation {
  name = "nix-build-uncached";

  src = pkgs.fetchurl {
    url ="https://github.com/Mic92/nix-build-uncached/releases/download/v0.1.0/nix-build-uncached_linux_amd64.gz";
    sha256 ="1l121il4fyx8swmmh0mkd9r9k2q29pvczipxlalnlq60s9yfmn4x";
  };

  phases = ["installPhase" "postInstall" "patchPhase"];

  nativeBuildInputs = [ pkgs.makeWrapper ];

  installPhase = ''
    mkdir -p $out/bin
    cp $src $out/bin/nix-build-uncached
    chmod +x $out/bin/nix-build-uncached
  '';

  postInstall = ''
    wrapProgram $out/bin/nix-build-uncached \
      --prefix PATH ":" ${pkgs.lib.makeBinPath [ pkgs.nix ]}
  '';
}

I'm getting things that look like this:

/home/felixmulder/.nix-profile/bin/nix-build-uncached: line 3: /nix/store/7db8jvm3sgwgpfgqnx571im4zm23bl2x-nix-build-uncached/bin/.nix-build-uncached-wrapped: cannot execute binary file: Exec format error
/home/felixmulder/.nix-profile/bin/nix-build-uncached: line 3: /nix/store/7db8jvm3sgwgpfgqnx571im4zm23bl2x-nix-build-uncached/bin/.nix-build-uncached-wrapped: Success

Do you have any instructions on how to start using this? :)

error: attribute '' missing

Hey!

Was trying to use this in one of my projects, and I end up getting this error:

error: attribute '' missing, at /tmp/295224101.nix:6:1

Here's a small repro that causes this for me:

# foo.nix, ran with `nix run nixpkgs.nix-build-uncached -c nix-build-uncached ./foo.nix`
with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "some-pkg";
  version = "0.0.1";
  src = ./.;

  dontBuild = true;
  installPhase = ''
    touch $out
  '';
}

I managed to get my hands on the generated tmp file which has the following contents:

{...} @args:
let
  set' = import "/home/anmonteiro/projects/lol/nix/foo.nix";
  set = if builtins.isFunction set' then set' args else set';
in [
set.""
]

nothing is being built

Skimming the source, I'm wondering if this change is related, somehow:

NixOS/nix@da8aac6

and:

nix-build-uncached/build.go

Line 44 in 2d17b31

if strings.HasPrefix(line, "these derivations will be built:") {

+ nix-build-uncached -build-flags '--out-link '\''/tmp/tmp.KRKvAjCnmR/result'\''   --option '\''extra-binary-caches'\'' '\''https://cache.nixos.org https://nixpkgs-wayland.cachix.org'\''  
 --option '\''trusted-public-keys'\'' '\''cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixpkgs-wayland.cachix.org-1:3lwxaILxMRkVhehr5StQprHdEo4IrE8sRho9R9HOLYA='\''   --o
ption '\''build-cores'\'' '\''0'\''   --option '\''narinfo-cache-negative-ttl'\'' '\''0'\''' packages.nix                                                                                   
$ nix-build --dry-run packages.nix --out-link /tmp/tmp.KRKvAjCnmR/result --option extra-binary-caches https://cache.nixos.org https://nixpkgs-wayland.cachix.org --option trusted-public-key
s cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixpkgs-wayland.cachix.org-1:3lwxaILxMRkVhehr5StQprHdEo4IrE8sRho9R9HOLYA= --option build-cores 0 --option narinfo-cache-neg
ative-ttl 0                                                                                                                                                                                 
warning: Git tree '/home/cole/code/nixpkgs-wayland' is dirty                                                                                                                                
these 82 derivations will be built:                                                                                                                                                         
  /nix/store/wzv2p515rgh0aiwksk291a40wjila8gc-source.drv                                                                                                                                    
  /nix/store/1k1zn6cysdi9n6g4zsmfp3fxmmrwppmg-wtype-74071228dea4047157ae82960a2541ecc431e4a1.drv                                                                                            
  /nix/store/1rld967l8xnzjki8gdshx429gjkll7m7-hg-archive.drv     
  ...
  /nix/store/xc7aif05vwqsl1js5lhkvavi54546658-emacs-pgtk-0b06fbaeb8f9cad185a452e44943a411d9bdedda.drv                                                                                       
  /nix/store/zwma1p2r2aagd4zgviz9rlssha7h81vm-freerdp-45a1f9cc1999ba9195d6fbe9a3f1f8feb273245d.drv                                                                                          
these 432 paths will be fetched (691.29 MiB download, 3458.52 MiB unpacked):                                                                                                                
  /nix/store/02xr4yq8svlg0vc2nzkbrw1s4sg4bk38-gtk+3-3.24.21                                                                                                                                 
  /nix/store/04158wjim7ngfq9bbvxg9jj4jlqsgww6-cups-2.3.3-dev                                                                                                                                
  /nix/store/04rgblqzcncl8xv9yclv11sh1f6qm9a8-libXt-1.2.0-dev                                                                                                                               
  ...
  /store/zxik0ry1lm16bm1dgqnmdbi769x5spbq-scdoc-1.11.0

# nothing builds

Output can't be piped to cachix

It looks like this prints stuff to stdout instead of stderr maybe.

It's nice to be able to just pipe everything to cachix, especially when still using nix-build and it might output numerous out-links, or results to stdout.

Is it just a matter of changing the fmt.Printlns in build.go to print to stderr instead?

Detection for '--experimental-features' has changed?

After yesterday's change, I tested locally, but hadn't tested with my CI builder that doesn't have daemon-wide experimental-features enabled.

Checked today, it hit another snag. The daemon doesn't have it enabled, the CLI is nixUnstable. But, you detect this and try to accomodate it. It looks like maybe the heuristic needs an update, due to behavior change in the CLI?

$ nix build --out-link /run/user/1000/tmp.DJ6KSQ7TNR/result --option extra-binary-caches https://cache.nixos.org https://nixpkgs-wayland.cachix.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixpkgs-wayland.cachix.org-1:3lwxaILxMRkVhehr5StQprHdEo4IrE8sRho9R9HOLYA= --option build-cores 0 --option narinfo-cache-negative-ttl 0 /nix/store/c9snhq4763cs50w9bapa2k88g9c1ac97-wofi-ac8dc17ab751.drv [snip]
error: --- Error --- nix
experimental Nix feature 'nix-command' is disabled; use '--experimental-features nix-command' to override
nix-build-uncached: nix build failed: exit status 1

I logged into my CI box, ran nix-shell for my project to load up the same env:

[nix-shell:~/nixpkgs-wayland]$ nix; echo $?
[prints usage]
0

[nix-shell:~/nixpkgs-wayland]$ nix build nixpkgs.hello
error: --- Error ---------------------------------------------------------------------------------- nix
experimental Nix feature 'nix-command' is disabled; use '--experimental-features nix-command' to override
1

So I think the detection needs to actually try something more aggressively?

`allowSubstitutes = false` issue

Derivations with allowSubstitutes = false like all the trivial builders are getting marked for rebuild every time.

nix-build-uncached doesn't build anything with nix 2.15

With the release of Nix 2.15 (see the release notes), this PR was merged: NixOS/nix#7600. As nix-build-uncached calls nix build /nix/store/hash-name.drv, only the derivations will be built but no outputs. With Nix 2.15 the new syntax nix build '/nix/store/hash-name.drv^*' must be used to actually build all outputs.

Do not truncate logs?

Since switching from nix-build to nix-build-uncached, it seems that build failures are truncated on our CI system to the last 10 lines. I am not sure if this is due to nix-build-uncached, or the way nix-build-uncached builds stuff. But it would be good to get full logs back. Is there an easy way to achieve that?

question about cachix's garbage collection

I'm using nix-build-uncached with cachix.

Since nix-build-uncached will skip building stuff already in my cachix cache, will cachix's garbage collection remove stuff I need since I didn't build/push them in the last runs?

Pass flags only to `nix build`

Currently, there is (as far as I'm aware) no way to pass flags only to the nix build.

When passing flags through -build-flags, these also get appended to the nix-build. Due to the syntax of these commands not being the same, it causes some issues for me. I want to access compilation logs, but currently I cannot pass the -L flag to nix build, as nix-build doesn't understand that flag.

Outputs existing in local store but not in cache are not built

I want to use nix-build-uncached on persistent builders, so that builds are even faster, because dependencies are already cached.

I have this script to build new outputs and upload them

      nix-build-uncached ./ci.nix -build-flags '-o results/x'
      if [ -e results/x ]; then
        nix -v sign-paths -r --key-file $NIX_CACHE_KEY ./results/x*
        nix copy --to $BINARY_CACHE_URL ./results/x*
      fi

Now if I mess up the signing & uploading part, I end up in a situation, where the outputs are present on the builder but not in the cache and nix-build-uncached will not create any outputs.

Unfortunately nix-build --dry-run does not provide any information about those paths.

Related NixOS/nix#1607

Flakes docs don't work

AFAICT the flakes docs don't work.

getFlake is only available when flakes is enabled, I don't think it works from nix-build, or at least I can't get it to.

I assume flake-compat will work? I'm going to try.

Using nix copy after nix-build-uncached

Hello,

Once we have build all the (uncached) packages defined in our overlays using nix-build-uncached,
we copy the packages to a directory using nix copy. This command sometimes fails because some package are not in the nix store (as it hasn't been build/fetched by nix-build-uncached).
Would it sound ok to add an option in nix-build-uncached to fetch the non build packages in the local nix store. We would then end up with the same packages in the /nix/store as if we would have run nix-build.

Thank you for this great tool ! All this should be a parameter to nix-build.