svanderburg/dysnomia

Dysnomia: A tool for deploying mutable components

dysnomia's Issues

How to force unlock?

While running disnixos-env, it tells me this:

[coordinator]: Acquiring locks...
[target: XX]: Acquiring a lock on profile: ...
...
Notifying lock on ...
...
Cannot lock service!
...
Notifying unlock on ...
[target: XX]: Cannot lock profile: ...

I tried a bunch of things but couldn't figure out how to force unlock the service. If I add --no-lock then the deploy proceeds as expected.

Of course, another question is how could the state end up like this. But I'm fine if I just figure out how to unlock 😅

How to model migrations?

I get that you can provision an "initial state", but usually this initial state is not just an available schema, but a series of migrations that lead up to the schema. Also, running systems might require migrations as well. How do I handle those?

[question] are there nix expression for build from sources?

Hi, thanks for great tools!

I need to build dysnomia from master branch to use systemd-unit module.

Tried to override nix expression from nixpkgs:

pkgs.dysnomia.overrideAttrs(oa: {
    name = "dysnomia";
    buildInputs = oa.buildInputs ++ [
      pkgs.autoreconfHook
    ];

    src = builtins.fetchGit {
      url = "https://github.com/svanderburg/dysnomia";
    };
    
    # also, tried to run ./bootstrap, the result is same
    preAutoreconf = ''
      ln -s README.md README
      mkdir -p config
    '';
  })

But got following error:

Making all in scripts
make[1]: Entering directory '/build/source/scripts'
Making all in default
make[2]: Entering directory '/build/source/scripts/default'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/build/source/scripts/default'
make[2]: Entering directory '/build/source/scripts'
false --output=dysnomia.1 --no-info --name 'Execute deployment operations of mutable components' "/nix/store/xfbmj7sl2ikicym9x3yq7cms5qx1w39k-bash-4.4-p23/bin/bash dysnomia"
make[2]: *** [Makefile:729: dysnomia.1] Error 1
make[2]: Leaving directory '/build/source/scripts'
make[1]: *** [Makefile:454: all-recursive] Error 1
make[1]: Leaving directory '/build/source/scripts'
make: *** [Makefile:363: all-recursive] Error 1
builder for '/nix/store/qkn7mq42zzb89svsrgb4kj0sl5s6l0kg-dysnomia.drv' failed with exit code 2
cannot build derivation '/nix/store/8zb9k5yw3pp0frsahvplshc47z006npg-dysnomia-fish-completions.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9p8kklhz7xq9nqman1x7rfd3ysaax5dg-home-manager-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zry6yi1jbjrvhz238psv9hr28ygmpwjd-home-manager-generation.drv': 1 dependencies couldn't be built
error: build of '/nix/store/zry6yi1jbjrvhz238psv9hr28ygmpwjd-home-manager-generation.drv' failed

Install on Amazon Linux

After ./bootstrap , ./configure , and make:

Making all in scripts
make[1]: Entering directory `/home/ec2-user/dysnomia/scripts'
false --output=dysnomia.1 --no-info --name 'Execute deployment operations of mutable components' "/bin/sh dysnomia"
make[1]: *** [dysnomia.1] Error 1
make[1]: Leaving directory `/home/ec2-user/dysnomia/scripts'
make: *** [all-recursive] Error 1

How to add a custom module when using disnixos + nixops?

Hi!

I want to make some changes to the postgresql-database.in module. I figured the easiest would be to copy it, modify it, and ship it with another name so dysnomia could see the new module. But I fail to do that in at least two ways:

1. All the @var@ variables do not get substituted correctly (especially the #!/bin/bash at the top).
2. I don't think dysnomia sees the module I copied over using nixops

I can share my full config if needed but here are the relevant parts:

In network.nix:

{
  myserver = { pkgs, ... }:
    let
      dysnomiaExtraModulePath = "dysnomia-modules";
    in
    rec {
      dysnomia = {
        enable = true;
        enableLegacyModules = false;
        extraModulePaths = ["/etc/${dysnomiaExtraModulePath}"];
      };

      environment.etc = {
        "${dysnomiaExtraModulePath}/postgresql-database-secret" = {
          source = ./dysnomia-modules/postgresql-database-secret.in;
          # mode = "0777";
        };
      };
    };
}

For the purpose of this issue, there's no need to show the specific changes to the dysnomia module I made, the issue is relevant with a verbatim copy of any module.

Issue about replacing shebang and variables

Afterwards I deploy using nixops:

$ nixops deploy -d vboxtest

And can see the file but none of the variables were replaced (which makes sense):

$ nixops ssh myserver
$ head /etc/dysnomia-modules/postgresql-database-secret -n 1
#!/bin/bash

Looking at an official module, the shebang gets replaced correctly:

$ nixops ssh myserver
$ head /nix/store/9d7jhh7xsyyx933blk0c504qff609qkl-dysnomia-0.10.1/libexec/dysnomia/postgresql-database -n 1
#!/nix/store/bm7jr70d9ghn5cczb3q0w90apsm05p54-bash-5.1-p8/bin/bash

I'm guessing I'm missing some call to one of the substitute* functions but I tried to grep in your repos and I don't see where that happens.

Issue about not finding module

The second issue comes from not being able to locate the module. At least that's what it looks like but I know the bash error messages can be cryptic and can hide a permission issue as something else for example.

Anyway, deploying returns this error:

$ disnixos-env -s services.nix -n network.nix -d distribution.nix --use-nixops
[...]
[coordinator]: Executing activation of services:
[target: myserver]: Activating service with key: 6c7779e925f58420b6761b4b04b8ada3412f8fcc8071ea2bc823dd943d5f6544 and package: /nix/store/47l1mkclinkqwfwgwd9r1i95vk1msrfy-ttrss with module: postgresql-database-secret in container: postgresql-database-secret, arguments: 
env: ‘/etc/dysnomia-modules/postgresql-database-secret’: No such file or directory
[target: arsenic]: Activation failed of service: 6c7779e925f58420b6761b4b04b8ada3412f8fcc8071ea2bc823dd943d5f6544
[...]

Although it looks like it should be able to find the module correctly:

$ nixops ssh myserver
$ echo $DYSNOMIA_MODULES_PATH 
/etc/dysnomia-modules:/etc/dysnomia/modules

I grepped github for any clue on how to create a new module but couldn't find anything useful. Can you give me pointers on how to achieve this?

Documentation not mentioning the need to explicitly add /etc/systemd-mutable

Looks like since this PR dysnomia fails activating a systemd-unit. The fix is to use boot.extraSystemdUnitPaths = [ "/etc/systemd-mutable/system" ]. But there's no mention of that in the manual and the only references are in tests . I don't know the best place to mention the need for this but I'd be happy to contribute a PR for updating the docs.

Use of localSystem

The dysnomia module (both the one from this repo, and the one in upstream nixpkgs, which is broken currently though) uses config.nixpkgs.localSystem which is probably not a good idea (see NixOS/nixpkgs#49765).
What is the value of localSystem used for anyway?

/etc/systemd-mutable/user will soon be gone from NixOS

Not sure if the path is still used here, but I thought I would drop a notice here: NixOS/nixpkgs@4d2437d

Adding a custom module to Dysnomia via a nixpkgs overlay

NOTE: The nixpkgs derivation for Dysnomia isn't in this repo, but I'm creating the issue here as the changes I would suggest considering include changes here too.

Firstly, the nixpkgs Dysnomia derivation's src is the pre-built release tarball from github instead of an actual git revision and while autoconf from stdenv is present, automake and autoreconfHook are not, so when someone patches Dysnomia to add a custom module to Dysnomia, Dysnomia in nixpkgs doesn't know how to build itself and the person has to also override the nativeBuldInputs to add the autoreconfHook, autoconf, automake, and help2man.

Example of a partial overlay overriding Dysnomia:

self: super:
{
  /*
  A hackish derivation that:
  1. Write a patch adding multiple dysnomia modules to Dysnomia to $out based on dysnomia's sources (not the release tarball)
  2. Sets passthru.buildInputs to the buildInputs that need to be added to Dysnomia alongside the patch.
  */
  dysnomia-modules = ... ;

  dysnomia = super.dysnomia.overrideAttrs (
    finalAttrs: previousAttrs: {
      nativeBuildInputs = (super.lib.lists.optional (previousAttrs ? nativeBuildInputs) previousAttrs.nativeBuildInputs) ++ [
        super.autoconf
        super.automake
        super.autoreconfHook
        super.help2man
      ];
      buildInputs = previousAttrs.buildInputs ++ [self.dysnomia-modules.buildInputs];
      patches = (super.lib.lists.optional (previousAttrs ? patches) previousAttrs.patches) ++ [
        self.dysnomia-modules
      ];
    }
  );
}

• Please consider changing the nixpkgs Dysnomia derivation to be able to reconfigure itself when patched so that people can rely on patching as one route to seamlessly inject custom modules when working from nixpkgs overlays or equivalent.

Adding custom modules through an overlay by writing a derivation that itself writes a patch file was not my first choice.
As the custom modules are bash scripts, they source the util script and effectively have a run-time dependency on the dysnomia package for that sole reason unless packaged by the dysnomia package.
Attempts to package the modules as normal derivations, not patches, meant dysnomia was in their buildInputs which understandably complicates any attempt to override dysnomia in an overlay to include them as buildInputs and to patch its installPhase to bring them along for the ride.
I only resorted to patches after export DYSNOMIA_MODULES_PATH=DIR did not actually result in Disnix+Dysnomia discovering the custom modules, though I did not troubleshoot in any depth what went wrong with my attempt at setting DYSNOMIA_MODULES_PATH and it could have been an error on my part perhaps.

Example of derivation hardcoded to write a patch for two Dysnomia custom modules named my-module and my-other-module.

{
diffutils
,dysnomia-src
,lib
,shellcheck
,stdenv
,some-runtime-dependency
}:
stdenv.mkDerivation
rec {
pname = "dysnomia-modules";
version = "main";
srcs = [dysnomia-src ./source-for-some-module ./source-for-another-module];
sourceRoot = "src";

/*
Assuming patch flags `-p1`, then diff should be run from the unpack directory, so create a symlink to Dysnomia's source as a sibling to the directory where the modified source is prepared.
*/
unpackPhase =
''
runHook preUnpack

_srcs=( ''${srcs:?} )

ln -s "''${_srcs[0]:?}" ./dysnomia
cp --reflink=auto -r --no-preserve=all -- "''${_srcs[0]:?}" ./${sourceRoot}
cp --reflink=auto --no-preserve=all -- "''${_srcs[1]:?}/my-module.sh" ./src/dysnomia-modules/my-module.in
cp --reflink=auto --no-preserve=all -- "''${_srcs[1]:?}/snapshot.nix" ./src/dysnomia-modules/snapshot.nix
cp --reflink=auto --no-preserve=all -- "''${_srcs[2]:?}/my-other-module.sh" ./src/dysnomia-modules/my-other-module.in

runHook postUnpack
'';

dontConfigure = true;

nativeBuildInputs = [diffutils];

buildInputs = [some-runtime-dependency];

/*
In the interests of not duplicating the patch-building boilerplate and not having to troubleshoot any possible conflicts in ambiguity of patch application, all modules share one derivation that produces one patch to add them in.
*/
buildPhase =
''
runHook preBuild

sed -Eie '/^[[:blank:]]*AC_OUTPUT[[:blank:]]*/ d' configure.ac
>>configure.ac cat <<EOF
AC_CONFIG_FILES([
dysnomia-modules/my-module
dysnomia-modules/my-other-module
])
AC_OUTPUT
EOF

>>dysnomia-modules/Makefile.am cat <<EOF
libexec_SCRIPTS += my-module my-other-module
EXTRA_DIST += my-module.in my-other-module.in
pkgdata_SCRIPTS += snapshot.nix
EOF

substituteInPlace dysnomia-modules/my-other-module.in \
	--subst-var-by useful-thing.pl '${some-runtime-dependency}/share/some-runtime-dependency/useful-thing.pl'

runHook postBuild
'';

doCheck = true;
checkInputs = [shellcheck];

checkPhase=''
runHook preCheck

shellcheck -- "''${_srcs[1]:?}/my-module.sh" "''${_srcs[2]:?}/my-other-module.sh"
shellcheck -- dysnomia-modules/my-module.in dysnomia-modules/my-other-module.in

runHook postCheck
'';

installPhase =
''
runHook preInstall

test ! -e "$out"

cd ..

# Expected exit status '1' because inputs should differ.
diff --minimal --new-file --recursive --text --unified -- dysnomia ${sourceRoot} >"$out" && exit 64 || test 1 = "$?"
test -f "$out"

cd ./${sourceRoot}

runHook postInstall
'';

doInstallCheck = true;

installCheckPhase =
''
runHook preInstallCheck

cd ../dysnomia

if ! patch --dry-run -p1 <"$out"
then
	echo "$out"
	cat "$out"
	exit 64
fi

cd ../${sourceRoot}

runHook postInstallCheck
'';

passthru = { inherit buildInputs; };

meta.description = "Patch for Dysnomia adding custom modules for ...";
}

• Please consider how the configure script and the nixpkgs derivation could be modified to make adding custom modules via nixpkgs overlays not require writing patch files like this.