numtide/nix-filter

a small self-contained source filtering lib

nix-filter - a small self-contained source filtering lib

STATUS: beta

When using nix within a project, developers often use src = ./.; for a project like this:

{ stdenv }:
stdenv.mkDerivation {
  name = "my-project";
  src = ./.;
}

This works but has an issue; on each build, nix will copy the whole project source code into the /nix/store. Including the .git folder and any temporary files left over by the editor.

The main workaround is to use either builtins.fetchGit ./. or one of the many gitignore filter projects but this is not precise enough. If the project README changes, it should not rebuild the project. If the nix code changes, it should not rebuild the project.

This project is a small library that makes it easy to filter in and out what files should go into a nix derivation.

Example usage

nix-filter works best for projects where the files needed for a build are easy to match. Using it with only an exclude argument will likely not reduce the reasons for rebuilds by a lot. Here's an Example:

{ stdenv, nix-filter }:
stdenv.mkDerivation {
  name = "my-project";
  src = nix-filter {
    root = ./.;
    # If no include is passed, it will include all the paths.
    include = [
      # Include the "src" path relative to the root.
      "src"
      # Include this specific path. The path must be under the root.
      ./package.json
      # Include all files with the .js extension
      (nix-filter.matchExt "js")
    ];

    # Works like include, but the reverse.
    exclude = [
      ./main.js
    ];
  };
}

Usage

Import this folder. Eg:

let
  nix-filter = import ./path/to/nix-filter;
in
 # ...

The top-level is a functor that takes:

• root of type path: pointing to the root of the source to add to the /nix/store.
• name of type string (optional): the name of the derivation (defaults to "source")
• include of type list(string|path|matcher) (optional): a list of patterns to include (defaults to all).
• exclude of type list(string|path|matcher) (optional): a list of patterns to exclude (defaults to none).

The include and exclude take a matcher, and automatically convert the string and path types to a matcher.

The matcher is a function that takes a path and type and returns true if the pattern matches.

The flake usage is like this:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-22.11";
    nix-filter.url = "github:numtide/nix-filter";
  };
  outputs = { self, nixpkgs, nix-filter }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
      # Avoid calling it nix-filter as it may result in an infinite recursion
      filter = nix-filter.lib;
     # ....
    in
    {
      # ...
    };
}

Builtin matchers

The functor also contains a number of matchers:

• nix-filter.matchExt: ext -> returns a function that matches the given file extension.
• nix-filter.inDirectory: directory -> returns a function that matches a directory and any path inside of it.
• nix-filter.isDirectory: matches all paths that are directories

Combining matchers

• and: a -> b -> c combines the result of two matchers into a new matcher.
• or_: a -> b -> c combines the result of two matchers into a new matcher.

NOTE: or is a keyword in nix, which is why we use a variation here.

REMINDER: both, include & exlude already XOR elements, so or_ is not useful at the top level.

Design notes

This solution uses builtins.path { path, name, filter ? path: type: true } under the hood, which ships with Nix.

While traversing the filesystem, starting from path, it will call filter on each file and folder recursively. If the filter returns false then the file or folder is ignored. If a folder is ignored, it won't recurse into it anymore.

Because of that, it makes it difficult to implement recursive glob matchers. Something like **/*.js would necessarily have to add every folder, to be able to traverse them. And those empty folders will end-up in the output.

If we want to control rebuild, it's important to have a fixed set of folders.

One possibility is to use a two-pass system, where first all the folders are being added, and then the empty ones are being filtered out. But all of this happens at Nix evaluation time. Nix evaluation is already slow enough like that.

That's why nix-filter is asking the users to explicitly list all the folders that they want to include, and using only an exclude is not recommended.

Future development

Add more matchers.

Related projects

• nixpkgs' lib.cleanSourceWith.
• All the git-based solutions. See https://github.com/hercules-ci/gitignore.nix#comparison

License

Copyright (c) 2021 Numtide under the MIT.

nix-filter's Issues

Support exact filer matching

I'd like to be able to specify exact files such as

rec { 
  artifact_a = nix-filter {
     root = ../../;
     include = [
        ./relative/path/to/a.ext
        ./another/path/to/b.ext
     ];
  };
  
  artifact_b = nix-filter {
     root = ../../;
     include = [
        artifact_a
        ./third/path/to/c.ext
     ];
  };
}

And i'd expect the output for artifact_b to contain

relative/path/to/a.ext,
another/path/to/b.ext,
third/path/to/c.ext,

keeping the directory heirarchy.

Why?

It would allow the user to composable build src lists. Artifact A needs files x, y, artifact B needs A + file z, whilst maintaining their relative paths. Unsure how to achieve file level granularity currently

matchExt never matches top-level files

Describe the bug

To Reproduce

Steps to reproduce the behavior:

1. setup filter:

nix-filter {
    # ...
    exclude = [
      (nix-filter.matchExt ".nix")
    ];
};

2. add a new line to the flake.nix
3. notice that a rebuilt of the package is triggered

Expected behavior
Adding a new line to the flake.nix should not trigger a re-build

System information

Additional context

How to use nix-filter for a path returned by gitignore.nix ?

I'd like to ignore both everything that's in my .gitignore with gitignore.nix and afterwards exclude more files with nix-filter.

I tried:

        src = nix-filter {
          root = gitignoreSource ./.;
        };

But that failed with:

error: assertion '((builtins).isPath  root)' failed

At:

nix-filter/default.nix

Line 20 in 3c9e33e

assert builtins.isPath root;

And I also tried:

src = nix-filter {
  root = builtins.path {
    path = gitignoreSource ./.;
  };

But that failed with:

error: string '/nix/store/r1r5ibqn3rh3mklzjb8l196ibxz42rrv-source' cannot refer to other paths

I also tried to first use gitignoreSource and afterwards nix-filter, but that failed similarly.

I noticed there's this open issue and I tried to help that by exposing a makeFilter function from this library, but I noticed all functions rely upon an argument root which is the full path to the source.

Any ideas?

Adding support for recursive file filtering without including empty directories.

I came up with an approach to fix the empty directory problem. I wanted to make you aware of it in case you'd be able to use it to improve the UX.
NixOS/nixpkgs#188301.

#27 breaks when using flakes with error: access to absolute path '/nix/store' is forbidden

Describe the bug

Since #27 this seems to break with flakes. If you add a check like

       ...
        nix-src = nix-filter.lib.filter {
          root = ./.;
          include = [
             (nix-filter.lib.matchExt "nix")
          ];
        };
       ...

        checks = flake-utils.lib.flattenTree {

          nixpkgs-fmt = pkgs.runCommand "check-nixpkgs-fmt"
            {
              buildInputs = with pkgs; [
                nixpkgs-fmt
              ];
            }
            ''
              mkdir $out
              # nixpkgs-fmt --check ${nix-src}
            '';
      ...

And then try to run nix flake check it breaks with:

nix flake check --print-build-logs --show-trace
warning: Git tree '/Users/dbarroso/workspace/nhost/be' is dirty
error: access to absolute path '/nix/store' is forbidden in pure eval mode (use '--impure' to override)

       … while realising the context of path '/nix/store'

       at /nix/store/klvk2xb6g11rkfpf1y10aln7jyd23a38-source/default.nix:156:9:

          155|     builtins.pathExists p
          156|     && (builtins.readDir (builtins.dirOf p)).${builtins.baseNameOf p} == "directory";
             |         ^
          157| }

       … while evaluating '_pathIsDirectory'

       at /nix/store/klvk2xb6g11rkfpf1y10aln7jyd23a38-source/default.nix:154:22:

          153|   # Returns true if the path exists and is a directory and false otherwise
          154|   _pathIsDirectory = p:
             |                      ^
          155|     builtins.pathExists p

       … from call site

       at /nix/store/klvk2xb6g11rkfpf1y10aln7jyd23a38-source/default.nix:20:14:

           19|     }:
           20|       assert _pathIsDirectory root;
             |              ^
           21|       let

       … while evaluating 'filter'

       at /nix/store/klvk2xb6g11rkfpf1y10aln7jyd23a38-source/default.nix:8:5:

            7|   filter =
            8|     {
             |     ^
            9|       # Base path to include

       … from call site

       at /nix/store/bkmqpzqyyka3qjs1cs9x6r8pp9kpx5fh-source/flake.nix:23:19:

           22|
           23|         nix-src = nix-filter.lib.filter {
             |                   ^
           24|           root = ./.;

       … while evaluating the attribute 'buildCommand' of the derivation 'check-nixpkgs-fmt'

       at /nix/store/vns9ic8pgb1dx7cfmg3ivyxz73yn92nh-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while checking the derivation 'checks.aarch64-linux.nixpkgs-fmt'

       at /nix/store/nbkbs33bkw54f60kv5c5y48714l00dpw-source/flattenTree.nix:16:9:

           15|       (sum // {
           16|         "${pathStr}" = val;
             |         ^
           17|       })

       … while checking flake output 'checks'

       at /nix/store/nbkbs33bkw54f60kv5c5y48714l00dpw-source/default.nix:137:17:

          136|               {
          137|                 ${key} = (attrs.${key} or { })
             |                 ^
          138|                   // (appendSystem key system ret);

To Reproduce

See above.

Expected behavior

It should work

System information

Using 825abbb, if you revert to the previous commit it works fine.

Additional context

N/A

Can this be used to filter the contents of a flake's input?

I'm trying to make a flake.nix for cargo-generate. The following works:

{
  description = "cargo, make me a project";
  inputs = {
    cargo-generate = {
      flake = false;
      url = "github:cargo-generate/cargo-generate";
    };
    nci = {
      inputs.nixpkgs.follows = "nixpkgs";
      url = "github:yusdacra/nix-cargo-integration";
    };
    nix-filter.url = "github:numtide/nix-filter";
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };
  outputs = inputs:
    let
      name = "cargo-generate";
      nix-filter = import inputs.nix-filter;
      pkgs = common: packages:
        builtins.map (element: common.pkgs.${element}) packages;
    in inputs.nci.lib.makeOutputs {
      config = common: {
        cCompiler = { package = common.pkgs.clang; };
        outputs = {
          defaults = {
            app = name;
            package = name;
          };
        };
        runtimeLibs = pkgs common [ "openssl" ];
      };
      pkgConfig = common:
        let
          override = {
            buildInputs = pkgs common [ "openssl" "perl" "pkg-config" ];
          };
        in {
          ${name} = {
            app = true;
            build = true;
            depsOverrides = { inherit override; };
            overrides = { inherit override; };
            profiles = { release = false; };
          };
        };
      root = inputs.cargo-generate.outPath;
    };
}

But it also includes some files that are unnecessary to the actual build. I tried replacing root with the following instead:

      root = nix-filter {
        root = inputs.cargo-generate.outPath;
        include = [ "./Cargo.lock" "./Cargo.toml" "./README.md" "./src" ];
      };

But this fails with "error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/Cargo.toml': No such file or directory" when running nix build. Listing the contents of that directory also shows that it's empty.

Is it possible to use use nix-filter in this way? If not, what would be an alternative solution?

add runnable tests

The ./tests folder currently doesn't test anything. It just allows a user to easily manually test all the use-cases.

I think a simple bash script that compares the output with an expected list of files is probably enough.

$ nix-instantiate --eval --json --strict tests/
trace: unmatched path=/home/zimbatm/go/src/github.com/numtide/nix-filter/tests/fixture1/README.md type=regular
trace: unmatched path=/home/zimbatm/go/src/github.com/numtide/nix-filter/tests/fixture1/package.json type=regular
trace: unmatched path=/home/zimbatm/go/src/github.com/numtide/nix-filter/tests/fixture1/src type=directory
{"all":"/nix/store/rgjgr6sz431l4ihw3dsqmx2p2s0nyf4c-source","combiners":"/nix/store/pf6i1czhnjjqiwv5hzc63i4gaizd75sm-source","trace":"/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source","with-inDirectory":"/nix/store/shkhc9pys5hpqxas1pvmghaard0lz1jw-source","with-inDirectory2":"/nix/store/kg31s34xq2cl2mgzai0409izqhdr6hyf-source","with-matchExt":"/nix/store/d4y4y9js500qh8cpq3y0wf6gds66hc2z-source","with-matchExt2":"/nix/store/gfjb5mhq3jscwap4imxhcdyk5f4wccl3-source","without-readme":"/nix/store/zs42mv8ks2sfrnh3l99fk04a3xnphjyq-source"}

This gives all the attributes and store paths.

Provide an alias for `or`

or, when used alone, seems to be a keyword. So, it's not possible to use the builtin or matcher inside a with statement:

  src = nix-filter {
    root = ./.;
    include = with nix-filter; [
      (and
        (or (inDirectory "src") (inDirectory "test"))
        (matchExt "hs"))
    ];
  };

fails with:

error: syntax error, unexpected OR_KW

Although there is a workaround of using nix-filter.or, it would be nice to provide an alias, so something like above is possible. The first name I'd reach in this case would probably be or_.

I do not know why this isn't the case with and.

matchExt doesn't match files in subfolders

Describe the bug

If I do:

        nix-src = nix-filter.lib.filter {
          root = ./.;
        };

I can see all the files, subfolders, files in subfolders, etc, being matched correctly. However, if I do:

        nix-src = nix-filter.lib.filter {
          root = ./.;
          include = with nix-filter.lib;[
             (matchExt "nix")
          ];
        };

I only see the *.nix files in the root folder being matched, none of the files with that extension in subfolders is matched at all.

To Reproduce

See above.

Expected behavior

Files with the specified extension in subfolders that would be matched otherwise should be matched too.

System information

Using commit 825abbb

Additional context

N/A

Typo in description

The description contains a typo: "container" instead of "contained".

can this be composed with gitignore based filter?

I'd like to have an explicit whitelist specified with nix-filter combined with a blacklist based on .gitignore. Can I compose nix-filter somehow with, for example: https://github.com/hercules-ci/gitignore.nix? Thanks!

Comparison with `cleanSourceWith`

I'm currently using cleanSourceWith like:

      let srcFilter = path: type:
        let
          p = baseNameOf path;
        in
          !(
            # ignore CI directories
            (type == "directory" && (p == ".github" || p == "ci")) ||
            # ignore CI files
            p == ".travis.yml" || p == "cloudbuild.yaml" ||
            # ignore flake.(nix|lock)
            p == "flake.nix" || p == "flake.lock" ||
            # ignore docker files
            p == ".dockerignore" || p == "docker-compose.yml" ||
            # ignore misc
            p == "rustfmt.toml"
          );

         # ...

          src = pkgs.lib.cleanSourceWith {
            src = ./.;
            filter = srcFilter;
            name = "foo-source";
          };

What is the advantage of using nix-filter?

Improve ergonomics for glob matching

One thing I noticed when using this on a real-world project, is that matchExt is a bit too wide.

What I mean is that typically I want to match under a specific folder. Eg: frontend/**/*.js. Not for the whole project. So matchExt doesn't scale really well with larger source bases.

Improve the doc

The doc could be better.

What is the best notation to describe nix functions?

Are all the use-cases covered in the readme?

Include with a directory doesn't include files within or recursive directories

Describe the bug

From reading the description I would expect including a directory (relative to the root) would include everything within it (excluding whatever is in the excludes)

To Reproduce

I am in a repository with a bin folder with some files:

❯ tree bin | head
bin
├── api_codes
├── api_codes.js
├── btest
├── bulk-decaffeinate
├── bundler_config

I then have the following:

  src = nix-filter
    {
      # apply clean source to remove .git
      root = ./.;
      include = [
        "bin"
      ];
      exclude = [
        (nix-filter.matchExt "class")
        (nix-filter.matchExt "jar")
      ];
    };

  buildPhase = ''
    ${tree}/bin/tree
  '';

❯ nix-build build.nix
this derivation will be built:
  /nix/store/i3jq894xcpcyh88q188nym5rvb1mxgfg-test.drv
building '/nix/store/i3jq894xcpcyh88q188nym5rvb1mxgfg-test.drv'...
unpacking sources
unpacking source archive /nix/store/1pjir24lijwsy86j6syq527yy9x51mq1-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
.
`-- bin

1 directory, 0 files`

Expected behavior

I would expect to see the bin folder with all it's contents.

System information

❯ niv show
nix-filter
  homepage: 
  url: https://github.com/numtide/nix-filter/archive/3e81a637cdf9f6e9b39aeb4d6e6394d1ad158e16.tar.gz
  owner: numtide
  branch: master
  url_template: https://github.com/<owner>/<repo>/archive/<rev>.tar.gz
  repo: nix-filter
  type: tarball
  sha256: 1gqbrsx9s7r3i324wi5w1il6xmfma24g8hvfgpbgyjvzpci8m30k
  description: a small self-container source filtering lib
  rev: 3e81a637cdf9f6e9b39aeb4d6e6394d1ad158e16