nix-community/haumea

Filesystem-based module system for Nix [maintainer=@figsoda]

Haumea

Filesystem-based module system for Nix

Haumea is not related to or a replacement for NixOS modules. It is closer to the module systems of traditional programming languages, with support for file hierarchy and visibility.

In short, haumea maps a directory of Nix files into an attribute set:

From	To
├─ foo/ │ ├─ bar.nix │ ├─ baz.nix │ └─ __internal.nix ├─ bar.nix └─ _utils/ └─ foo.nix	{ foo = { bar = <...>; baz = <...>; }; bar = <...>; }

Haumea's source code is hosted on GitHub under the MPL-2.0 license. Haumea bootstraps itself. You can see the entire implementation in the src directory.

Why Haumea?

• No more manual imports

  Manually importing files can be tedious, especially when there are many of them. Haumea takes care of all of that by automatically importing the files into an attribute set.

• Modules

  Haumea takes inspiration from traditional programming languages. Visibility makes it easy to create utility modules, and haumea makes self-referencing and creating fixed points a breeze with the introduction of self, super, and root.

• Organized directory layout

  What you see is what you get. By default¹, the file tree will look exactly like the resulting attribute set.

• Extensibility

  Changing how the files are loaded is as easy as specifying a loader, and the transformer option makes it possible to extensively manipulate the tree.

➔ Getting Started

Footnotes

1. Unless you are doing transformer magic ↩

haumea's Issues

Infinite recursion when migrating my config

I am not sure whether this is a problem with how I originally set up my config, haumea, or a combination of the two. In certain spots, especially when referring to config I get infinite recursion just when migrating a line from my original config to a folder that haumea is loading. Oddly, this recurision is solved by lib.mkForce but it feels hacky.

Some examples:

Just specifying which package vscodium module should use.
Lyndeno/nix-config@4add99c

+ # FIXME: Why do I need mkForce to avoid recursion?
+ package = lib.mkForce pkgs.vscodium;

Referencing config to get the proper kernel.
Lyndeno/nix-config@c98095a

# Haumea file
+  # FIXME: For some reason mkForce is needed to prevent infinite recursion
+  kernelPackages = lib.mkForce config.boot.zfs.package.latestCompatibleLinuxPackages;

# Old file
-  boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

And in this case, not even referencing config, just changing the kernel.
Lyndeno/nix-config#dda0cc2

# FIXME: For some reason mkForce is needed to prevent infinite recursion
 kernelPackages = lib.mkForce pkgs.linuxPackages_rpi4; # Raspberry pies have a hard time booting on the LTS kernel.

In all these cases, the original line worked fine before migrating it to my haumea layout. I am not sure if maybe I am loading the folder in a non-ideal way, or if the way haumea works makes recursion more likely with the way I am configuring my system. The most puzzling cases to me are the ones where I am just changing the kernel package or the vscode package, not something I would expect to cause recursion. I am wondering if anyone has any idea how to fix these errors without resorting to mkForce.

Maybe this is not the right spot to post this problem, but I thought since I am running into this when converting my configs with haumea, maybe someone here could provide valuable insight.

Thank you for the very nice tool! It is allowing me to clean and organize my configs very nicely.

How to deal with attributes with '_' at beginning of name?

Besides using liftDefault?

For example, _1password-gui.

I will use default.nix for now just wondering if there is any other way to handle this?

Add "cursor" relative to import root to the loader API

Hi Everyone,

first of all, thank you for this library. It provides me with a simple solution for what has annoyed me the most since I've started writing my NixOS config about 4 weeks ago. During that time, I've been trying to find a more ergonomic way to manage NixOS module files and I think I now managed to find a decent way using Haumea. If you are at all interested, the dotfiles are available here, with a small preliminary write-up of how it works, but it's by no means required to understand what follows.

With that as context, now to my actual issue/proposal:
For the project I wrote a custom loader, which modifies the self, super and root attributes such that their functionality is preserved in a way that is actually useful in the context of NixOS modules. For this loader I'm actually only interested in a "cursor" relative to the root, so e.g. [ "base" "feature" ] for a file at <load path>/base/feature.nix. I then use this cursor together with nixpkgs.lib.getAttrFromPath to navigate the attribute tree.

However, the API currently passes only the absolute loading path to a loader, so e.g. /nix/store/<hash>-source/<load path>/base/feature.nix. This leads to the awkward situation that one needs to reverse engineer the cursor from this absolute path. In general, this is tricky, especially considering that the information is already available within the code of this library, namely in tree.pov in load.nix.

The original code of src/load.nix in lines 78-85 looks currently as follows:

{
   content = fix (self:
     (head matches).loader
       # inputs arg
       (inputs // {
         inherit self;
         super = getAttrFromPath tree.pov (view tree);
         root = view tree;
       }) 
       # path arg
       (src + "/${path}"));
}

My current patch to Haumea just adds a third argument to the API in load.nix:

{
   content = fix (self:
     (head matches).loader
       # inputs arg
       (inputs // {
         inherit self;
         super = getAttrFromPath tree.pov (view tree);
         root = view tree;
       })
       # path arg
       (src + "/${path}")
       # cursor arg
       (tree.pov ++ [name]));
}

There are also probably other ways of approaching this, but this one is the most straight forward in my opinion. Since I have the patch already, I could easily extend this to a pull request, but I would probably need some help with testing, as I currently don't use any other loaders than my own and the default one.

I realize this might not be worth the break in the API or even a use case you want to support. But for me it would go a long way to increase the ease of use in these edge cases where you only want to do some relative changes within the structure of the loaded files.

Thank you again for your work, best regards.

How to create attributes from variable path length suitable for callPackage?

Is there a way to do the same by just using a haumea transformer and/or loader?

# My package definitions are organized in the same way as in nixpkgs.
# Given /some/long/descriptive/path/to/packageName/default.nix
# I would like the resulting attributes to be
{ packageName = "/some/long/descriptive/path/to/packageName/default.nix"; }

At the moment, the following code does it for me

{ inputs, ... }:

let
  inherit (inputs) haumea;
  inherit (inputs.nixpkgs-lib.lib)
    foldl'
    attrNames
    concatStringsSep
    isPath
    isString
    isAttrs
    mapAttrs'
    nameValuePair
    last
    splitString;

  flattenTree = tree:
    let
      recurse = sum: path: val:
        foldl'
          (sum: key: op sum (path ++ [ key ]) val.${key})
          sum
          (attrNames val);

      op = sum: path: val:
        let
          pathStr = concatStringsSep "." path;
        in
        if (isPath val || isString val)
        then (sum // { "${pathStr}" = val; })
        else
          if isAttrs val
          then (recurse sum path val)
          else sum;
    in
    recurse { } [ ] tree;

  importPackages = src:
    let
      attrTree = haumea.lib.load {
        inherit src;
        loader = haumea.lib.loaders.path;
        transformer = [ (_: v: v.default or v) ];
      };

    in
    mapAttrs'
      (k: v: nameValuePair (last (splitString "." k)) v)
      (flattenTree attrTree);
in
importPackages

Move comments to the code

I feel similarly to https://discourse.nixos.org/t/haumea-filesystem-based-module-system-for-nix/26902/15, aka I have trouble making sense of the documentation. I find the readme great but the git book useless.
I had to look at @figsoda 's dotfiles to see how to use it.

I cloned the repo to get a better understanding and was surprised to see that the comments exist in the doc book but not in the code. Moving the doc to src/load.nix the doc for the load function with what it expects as input would help greatly IMO.

Use `haumea.lib.load` as functor for flake itself

This may be a bit of a controversial feature suggestion, but if we use the nix feature to declare __functor in attrs, we can make attribute sets callable. This way we can make the haumea flake itself callable. Since we have a go-to function that most people would want to be using with haumea.lib.load, we could make that the default function of the flake. If we write in the main flake:

outputs = { self, nixpkgs }: rec {
  # ...
  __functor = _system: args: self.lib.load args;
};

Then, in a using flake we could write:

outputs = { self, haumea, nixpkgs }: {
  lib = haumea {
    src = ./src;
    inputs = { inherit (nixpkgs) lib; };
  };
};

doc improvement: define a new type `Loader<NixExpr>`

Hi! I was trying to use a matcher and was confused by the

... ({ self, super, root, ... } -> Path -> a ) ...

everywhere in the doc. Only after I read the source did I realize that this is simply a loader. I believe it would be easier to define:

Loader<a> := { self, super, root, ... } -> Path -> a

... at the top of e.g. loaders and use that instead. Do you think this is a good idea? I am not sure if Loader<a> := is the nix way to annotate a new type, though; the angled brackets are simply borrowed from rust. Also, replacing some of the a with e.g. NixExpr would make it feel more descriptive to me.

If you all agree with this proposal but don't have time to implement this, I can try to submit a pull request, some time (I am very slow though, haha).

exclude specific directory names

Would it be possible to specify a list of directory names to be ignored while scanning?

hoist{Attrs,List}: infinite recursion encountered in evalModules

It seems that the judgment of introducing the value of config in hoisted will cause a problem of infinite recursion encountered

Notice: This problem is only triggered in evalModules and when the value of config for conditional judgment is introduced at the beginning of _imports(hoisted APIs).

Issue output:

       … while calling the 'head' builtin

         at /nix/store/bwps706g6ywl1811r7bwv32py12129ci-source/lib/attrsets.nix:780:11:

          779|         || pred here (elemAt values 1) (head values) then
          780|           head values
             |           ^
          781|         else

       … while calling the 'foldl'' builtin

         at /nix/store/yf26s7734ql7q05zlq6nq4y5q3rjyhj7-source/lib/trivial.nix:63:8:

           62|     let reverseApply = x: f: f x;
           63|     in builtins.foldl' reverseApply val functions;
             |        ^
           64|

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: infinite recursion encountered

       at /nix/store/yf26s7734ql7q05zlq6nq4y5q3rjyhj7-source/lib/modules.nix:261:21:

          260|           (regularModules ++ [ internalModule ])
          261|           ({ inherit lib options config specialArgs; } // specialArgs);
             |                     ^
          262|         in mergeModules prefix (reverseList collected);

Env:

• lib.evalModule
• haumea.load

 haumea.lib.load {
=   1       src = ./recipes/nixosModules;
= 101       inputs = modulesArgs;
=   1       transformer = with haumea.lib.transformers; [
=   2         liftDefault
=   3         (hoistLists "_imports" "imports")
=   4         (hoistAttrs "_options" "options")
=   5       ];
=   6     };

Since this problem is slightly confusing, I provided several causes that will help you understand easily.
Cases:

{config,lib}:

_imports =  ([
{
  # worked
  services.emacs.test = config.services.emacs.enable;
}
]);
#  error: infinite recursion encountered
_imports =  lib.optionals config.services.emacs.enable ([ {}
]);