gabriella439/haskell-nix

Nix and Haskell in production

Nix and Haskell in production

This guide documents how I use Nix for Haskell development. Feel free to open issues or pull requests if you would like to contribute or suggest improvements

The purpose of this project is to support two Haskell workflows:

• Workflow #1: Nix provisions the development environment
  • Nix provides all dependencies and the Haskell toolchain
  • You still build the root project using cabal (overview, user's guide)
  • This approach is ideal for development as it supports incremental builds
• Workflow #2: Nix builds the root project for you
  • This approach is ideal for continuous integration (especially Hydra)

The emphasis of this guide is to be as robust as possible and gracefully handle writing Haskell projects at scale. Some of the suggestions in this guide might be overkill for a small Haskell project but are essential when managing multiple private Haskell projects across a team of developers.

This guide is based partly on the Haskell section of the nixpkgs manual and partly on experience using Nix and Haskell in production at Awake Security.

Background

Nix is not a cabal replacement and Nix actually complements cabal quite well. Nix is much more analogous to a stack replacement. stack does provide some support for Nix integration, but this document does not cover that. Instead, this document describes how to use Nix in conjunction with cabal for Haskell development

The main benefits of using Nix over stack are:

• Binary caches

  Nix lets you download precompiled Hackage packages whereas stack compiles them on your computer the first time you depend on them

• Space efficiency

  stack creates a copy of each package for each resolver. This means that if you have two projects with different resolvers then they will not use the same copy of shared dependencies

• Generality

  Nix is a language-independent build tool. This means you can use Nix to also build and customize non-Haskell dependencies (like gtk). This uniform language simplifies build tooling and infrastructure.

• Larger ecosystem

  Nix provides a large ecosystem of tools that integrate with anything that Nix can build, such as Hydra (continuous integration), NixOS (an operating system), and NixOps (a deploy tool)

• Flexibility

  Nix is a powerful tool in the hands of advanced users. You can make very deep and sweeping changes to your toolchain, such as recompiling everything with security hardening

The main disadvantage of using Nix over stack are:

• Verbosity

  Nix derivations for Haskell projects are significantly more complex than their corresponding stack.yaml files. The release.nix files in this repository are the Nix analog of a stack.yaml file and you can see for yourself the increase in complexity as the examples progress in difficulty.

• Poor error messages

  Nix is an untyped language with no special Haskell integration, so error messages are unhelpful

• Nix cannot incrementally compile Haskell libraries

  Note that you can still use Nix to provision a development environment and incrementally compile a Haskell package using cabal. However, if you use Nix to build the package then Nix will build the package from scratch for every minor change. In theory, this could be fixed to have Nix directly support incremental Haskell builds but this has not been done yet.

• Worse user experience

  Nix does not provide many conveniences that stack does such as bootstrapping new projects or "file watch"

Both Nix and stack use curated package sets instead of version bounds for dependency management. stack calls these package sets "resolvers" whereas Nix calls these package sets "channels". Nix provides stable channels with names like NixOS-18.09 (analogous to stack's LTS releases) and then an unstable channel named nixpkgs-unstable (analogous to stack's nightly releases)

Related guides

• Nix Haskell Monorepo Tutorial - Guide on how to scale Nix development to a larger repository containing all of a company's internally-developed Haskell packages

Related tools

Before continuing, I'd like to mention some other tools for mixing Haskell with Nix:

• tinc - this uses cabal's solver to select which Haskell packages to use instead of the curated Haskell package set from nixpkgs
• styx - This tool provides a stack-like interface to managing Haskell dependencies using Nix
• haskell-overridez - Tool that automates dependency management as described in this guide

Setup

Before you begin, you must install Nix if you haven't already:

$ curl -L https://nixos.org/nix/install | sh

You must also install cabal2nix and nix-prefetch-git:

$ nix-env --install cabal2nix
$ nix-env --install nix-prefetch-git

You also need to install cabal if you haven't done so already. You can either use your installed cabal or you can use nix to install cabal for you:

$ nix-env --install cabal-install

Make sure that you have a fairly recent version of cabal installed since these examples will use GHC 8 which requires version 1.24 or later of cabal. You can check what version you have installed by running:

$ cabal --version

Finally, run cabal update if you haven't done so already

Organization

This tutorial is split into several tutorial projects in the project*/ subdirectories. Read the README.md file in each subdirectory in order to follow the tutorial:

• Project 0 - Nix basics
• Project 1 - Dependency management
• Project 2 - Non-Haskell dependencies
• Project 3 - Customizing Haskell projects
• Project 4 - Advanced dependency management

haskell-nix's Issues

Wrong source directory in project4

In project4/nix/project4.nix the source directory is specified as src = ./.. This causes nix-build to fail as follows:

project4 master $ nix-build --attr project4 release0.nix
these derivations will be built:
  /nix/store/ckyjcd2jg0ap23dc8jx8b47l1kscillq-project4-1.0.0.drv
[..]
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /tmp/nix-build-project4-1.0.0.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/c0pq35gpny9cr839j0vax9sv3bpq869k-project4-1.0.0 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --with-gcc=gcc --package-db=/tmp/nix-build-project4-1.0.0.drv-0/package.conf.d --ghc-option=-optl=-Wl,-rpath=/nix/store/c0pq35gpny9cr839j0vax9sv3bpq869k-project4-1.0.0/lib/ghc-8.0.2/project4-1.0.0 --ghc-option=-j1 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --ghc-option=-split-sections
Setup: No cabal file found.
Please create a package description file <pkgname>.cabal
builder for ‘/nix/store/ckyjcd2jg0ap23dc8jx8b47l1kscillq-project4-1.0.0.drv’ failed with exit code 1
error: build of ‘/nix/store/ckyjcd2jg0ap23dc8jx8b47l1kscillq-project4-1.0.0.drv’ failed

This is fixed with src = ../..

PR in #29

dhall-nix?

This may be more of a question than an issue. I noticed in the main README you mention that nix has worse error messages and that is owed to not having a type system. I'm wondering if you have used dhall-nix to address this. The syntax is so similar that it doesn't seem like it would add much overhead in terms of noise. Is the problem that you'd need to layer a build process (make?) over the whole project to convert your source-controlled dhall expressions into git-ignored nix expressions?

Error builing `project1`/`release1`

Hello, Gabriel! Thanks for your work on this guide!
I'm just starting with Nix + Haskell, so chances are I might be doing smth wrong.

While reading and trying out examples, I noticed that I can't build the project1/relase1:

haskell-nix/project1 [master●] » nix-build --attr project1.nix release1.nix
error: attribute 'nix' in selection path 'project1.nix' not found

Related tool: tinc

Not really an issue, but you might be interested in tinc, which has experimental support for instantiating the entire Cabal install plan via Nix. See https://github.com/sol/tinc/blob/nixpkgs/NIX.md. It's far from polished but still a small step. Make sure to use the nixpkgs branch from sol/tinc when trying this out (or install via haskellPackages.tinc from nixpkgs).

Project3 problem with config

Hi,

I have setup my release.nix code like the following:

let
  config = rec {
    allowUnfree = true;
    allowBroken = true;
    packageOverrides = pkgs: rec {
      haskellPackages = pkgs.haskellPackages.override {
        overrides = haskellPackagesNew: haskellPackagesOld: rec {
          frontend =
	    haskellPackagesNew.callPackage ./default.nix {};
	  };
      };
    };

    permittedInsecurePackages = [ "webkitgtk-2.4.11" ];
  };

  pkgs = import <nixpkgs> { inherit config; };

in
  { frontend = pkgs.haskellPackages.frontend; }

then in my shell.nix, i insert this code (import ./release.nix).frontend.env

However, when i run nix-shell -p haskell.compiler.ghcjs --command "cabal configure"

I got the following error:

error: Package ‘ghcjs-0.2.0’ in ‘/nix/store/wr91a710329n2jzsfd3biykyjwxv8pbn-nixpkgs-17.09pre102933.26625c9/nixpkgs/pkgs/development/compilers/ghcjs/base.nix:108’ is marked as broken, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

(use ‘--show-trace’ to show detailed location information)

the question is, i had already set the allowBroken = true; on my release.nix, but why it seems i have not set the allowBroken yet.

Does the approach is incorrect?

thank you.

`nix-build --attr project1 release2.nix` fails

nix-build --attr project1 release2.nix fails with this error message about a missing dependency.

Configuring optparse-applicative-0.13.0.0...
Setup: Encountered missing dependencies:
QuickCheck ==2.8.*
builder for ‘/nix/store/931j8l7jxfbv7rfmaamzxkk2jhqgwj2y-optparse-applicative-0.13.0.0.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/jffz98kpx92bzlzmdx1r0s56pigzaaga-turtle-1.3.2.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/p0w8qqjl2frcc5ip7cpr7iy7sfrrycmp-project1-1.0.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/p0w8qqjl2frcc5ip7cpr7iy7sfrrycmp-project1-1.0.0.drv’ failed

Changing the compiler to ghc784 help needed

Hi @Gabriel439,

using the apporach described here I tried to build flickr haskell.

Basically modifying to use the correct network-uri and mime versions works, but now I am stuck at the ghc-Version to use, because the flickr binding is using a now incompatible ("Make Applicative a superclass of Monad") implementation approach.

My hope was that specifying --argstr compiler ghc784 solved it but that basically lead me nowhere:

error: attribute 'ghc784' missing, at /home/vagrant/flickr/release5.nix:8:27

(the mentioned release5.nix being bascially the same as in 384e1d6 )

Seems a bit similar to #15, but I can't see it through.

Why both a default.nix and releaseX.nix?

Hi.
When I was going through the tutorial I was unsure of why there is both a default.nix and releaseX.nix.
A sentence describing the difference between these would be great!
I guess it has something do with releaseX.nix setting up nix infrastructure, whereas default.nix contains project build instructions.

https://github.com/Gabriel439/haskell-nix/blob/774325aeeba379f42d3e11ab824cc8c78876eb40/project0/README.md#nix-derivations

Why does this pick up the Haskell package called blas?

I tried to copy the example project1 but ran into this problem.

project1.cabal

name: project1
version: 1.0.0
license: BSD3
license-file: LICENSE
cabal-version: >= 1.18
build-type: Simple

executable project1
    build-depends: base < 5, hmatrix
    main-is: Main.hs
    default-language: Haskell2010

which via cabal2nix gives

hmatrix.nix

{ mkDerivation, array, base, binary, blas, bytestring, deepseq
, liblapack, random, semigroups, split, stdenv, storable-complex
, vector
}:
mkDerivation {
  pname = "hmatrix";
  version = "0.18.1.1";
  src = /Users/dom/hmatrix/packages/base;
  libraryHaskellDepends = [
    array base binary bytestring deepseq random semigroups split
    storable-complex vector
  ];
  librarySystemDepends = [ blas liblapack ];
  preConfigure = "sed -i hmatrix.cabal -e '/\\/usr\\//D'";
  homepage = "https://github.com/albertoruiz/hmatrix";
  description = "Numeric Linear Algebra";
  license = stdenv.lib.licenses.bsd3;
}

I copied release1.nix

let
config = {
  packageOverrides = pkgs: rec {
    haskellPackages = pkgs.haskellPackages.override {
      overrides = haskellPackagesNew: haskellPackagesOld: rec {
        project1 =
          haskellPackagesNew.callPackage ./default.nix { };
        # QuickCheck =
        #   haskellPackagesNew.callPackage ./QuickCheck.nix { };
        hmatrix =
          haskellPackagesNew.callPackage ./hmatrix.nix { };
      };
    };
  };
};

pkgs = import <nixpkgs> { inherit config; };

in
{ project1 = pkgs.haskellPackages.project1;
}

default.nix

{ mkDerivation, base, stdenv, hmatrix }:
mkDerivation {
  pname = "project1";
  version = "1.0.0";
  src = ./.;
  isLibrary = false;
  isExecutable = true;
  executableHaskellDepends = [ base hmatrix ];
  license = stdenv.lib.licenses.bsd3;
}

But I get

nix-build --attr project1 release1.nix -I /Users/dom/nixpkgs
these derivations will be built:
  /nix/store/nzsi0png5r2fpmffp41nnp29rh8n8q7z-blas-0.7.6.drv
  /nix/store/2x99czm9r889l0xkhnmmlh3kh4lf13rg-hmatrix-0.18.1.1.drv
  /nix/store/17dpxghi5jv7kcphxxprrvnykqqy687q-project1-1.0.0.drv
building path(s) ‘/nix/store/3w5d03war10ldilyr4f2g4f4crx7l18a-blas-0.7.6’, ‘/nix/store/m4zv23dnhq3l04wkhmxqmlbr47idigbz-blas-0.7.6-doc’
setupCompilerEnvironmentPhase
Build with /nix/store/2szhjd6206nmvl0fakkadai4jwfcdycc-ghc-8.0.2.
unpacking sources
unpacking source archive /nix/store/9fnnp9dqb3g407vgnzmyvbb63hmanj2c-blas-0.7.6.tar.gz
source root is blas-0.7.6
setting SOURCE_DATE_EPOCH to timestamp 1231969367 of file blas-0.7.6/tests/Vector.hs
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-blas-0.7.6.drv-0/package.conf.d -j1 -threaded
[1 of 1] Compiling Main             ( Setup.lhs, /private/tmp/nix-build-blas-0.7.6.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/3w5d03war10ldilyr4f2g4f4crx7l18a-blas-0.7.6 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/nix/store/m4zv23dnhq3l04wkhmxqmlbr47idigbz-blas-0.7.6-doc/share/doc --with-gcc=clang --package-db=/private/tmp/nix-build-blas-0.7.6.drv-0/package.conf.d --ghc-option=-optl=-Wl,-headerpad_max_install_names --ghc-option=-j1 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --extra-include-dirs=/nix/store/cfl3g7nakfffmk3a9g8frjbj74wfayra-libc++-4.0.1/include --extra-lib-dirs=/nix/store/cfl3g7nakfffmk3a9g8frjbj74wfayra-libc++-4.0.1/lib
Configuring blas-0.7.6...
Setup: Encountered missing dependencies:
QuickCheck >=1.2 && <2
builder for ‘/nix/store/nzsi0png5r2fpmffp41nnp29rh8n8q7z-blas-0.7.6.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/2x99czm9r889l0xkhnmmlh3kh4lf13rg-hmatrix-0.18.1.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/17dpxghi5jv7kcphxxprrvnykqqy687q-project1-1.0.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/17dpxghi5jv7kcphxxprrvnykqqy687q-project1-1.0.0.drv’ failed

Why is it trying to build the Haskell package called blas rather than just get the blas library?

Question about use of packageOverrides

In your examples the way you do overrides and adding project attributes (e.g. project1) is with packageOverrides which uses the config attribute when importing nixpkgs. However when you use config on import then overrides in ~/.config/nixpkgs/config.nix are no longer considered.

On one hand this is good since you want all configuration local to the project so there's no ambient state leaking in on your system. But on the other hand if you need say, a fork of some tooling (ghc-mod, intero, whatever) available in nix-shell then by using packageOverrides in your project you no longer get the tooling you need, unless you include it in your project build files which is kind of weird.

An alternative I have seen is to do overrides using something like:

overrideHaskellPkgs = self: super: {
  haskellPackages = myOverrides self super;
};

pkgs = fix' (extends overrideHaskellPkgs ...)

which is how haskellPackages does things in Nix I believe. This way you get overrides but also the ability to preserve your config.

Curious to hear your thoughts.

Attribute 'ghc801' missing at all-packages.nix

Hi @Gabriel439, thanks to your tutorial on adding dependencies, it works when i was dealing with missing them when i compile.

However, i don't know how to deal with the missing attribute, and why it happens.

I got the following error:

nix-build -A frontend release.nix
error: attribute ‘ghc801’ missing, at /nix/store/5kqp52f7p2mbhlpp8m7bs83j8wz0czw4-nixpkgs-channels-09c3d04b0e53d65f60569fc01698decff3a657a3-src/pkgs/top-level/all-packages.nix:4692:21
(use ‘--show-trace’ to show detailed location information)

here is my code;

{ compiler ? "ghcjs" }:

let
  cabal2nixResult = src: pkgs.runCommand "cabal2nixResult" {
    buildCommand = ''
      cabal2nix file://"${src}" > "$out"
    '';
    buildInputs = with pkgs; [ cabal2nix]; } "";

  sources = {
    gtk2hs = pkgs.fetchFromGitHub {
      owner = "gtk2hs";
      repo = "gtk2hs";
      rev = "eee61d84edf1dd44f8d380d7d7cae2405de50124";
      sha256 = "12i53grimni0dyjqjydl120z5amcn668w4pfhl8dxscjh4a0l5nb";
    };
  };

  config = rec {
    allowUnfree = true;
    allowBroken = true;
    packageOverrides = pkgs: with pkgs.haskell.lib; rec {
      haskell.packages.${compiler} = pkgs.haskell.packages.${compiler}.override {
        overrides = self: super: rec {

          gtk2hs-buildtools = self.callPackage (cabal2nixResult "${sources.gtk2hs}/tools") {};

          semigroups = pkgs.haskell.lib.addBuildDepends
                         super.semigroups (with self; [
			   hashable
			   tagged
			   text
			   unordered-containers
			 ]);

          tasty-ant-xml = self.callPackage ./tasty-ant-xml.nix {};          
	  
          frontend = self.callPackage ./default.nix {};
	 
          reflex   = self.callPackage ./reflex {};

          reflex-dom = self.callPackage ./reflex-dom {};
	};
      };
    };
    permittedInsecurePackages = [ "webkitgtk-2.4.11" ];
  };

  pkgs = import ./nixpkgs { inherit config; };

in
  { frontend = pkgs.haskell.packages.${compiler}.frontend; }

Changing the compiler doesn't work well with callCabal2nix

When using the suggested format

      haskell = pkgs.haskell // {
        packages = pkgs.haskell.packages // {
          "${compiler}" = pkgs.haskell.packages."${compiler}".override {
            overrides = galoisOverrides;
          };
        };
      };

to override the GHC version, the pkgs.haskellPackages attribute is not overriden, so e.g. callCabal2nix doesn't use the selected version. Additionally, one must select the package from the new compiler's package set:

project1 = pkgs.haskell.packages.${compiler}.project1;

Why not use the following instead?

      haskellPackages = pkgs.haskell.packages."${compiler}".override {
        overrides = [...];
      };

This provides a more consistent interface for the rest of your Nix files.

Explain how Haskell package sets and compiler versions are linked

I've been trying to convert a stack-based project to nix today and ran into a problem. Nix is generally presented as being similar to stack in that the packages are all guaranteed to build together. We're also told that haskellPackages is just a shortcut for writing haskell.packages.<compiler-version> and that we can switch to another compiler just by using a parameter here.

I assumed that switching to ghc821 (which I was already using with a stack nightly snapshot) would give me a set of packages that are compatible with this compiler, but this isn't the case. Specifically hasql has a test dependency on rebase-1.0.8 which has a bound on base < 4.10 and they won't build together. At this point I'm not really sure how the package set for this compiler (or indeed any) is determined. Is the Haskell package set only specific to one version of nixpkgs and only guaranteed to work with the default compiler? Or do you get a different Haskell package set if you switch compilers while still on the same version of nixpkgs?

The project 1 README seems to be where curated packages sets are discussed. Perhaps an explanation of where they come from and how (or if) they depend on the chosen compiler could be added here?

Define the word derivation

You use the word derivation without actually saying what one is.

Native runtime dependencies question

Does the native dependencies section applies to native runtime dependencies on executables ?

For instance I have a haskell executable that uses turtle under the hood. Turtle is going to call two external executable: jq and pepper.

Is there an idiomatic way to make sure the closure of the derivation will contain both executables (while using justStaticExecutables.

I have been trying this : http://nixpaste.lbr.uno/KdgmFXii?nix

Unfortunately when I try nix run -f release.nix project I will end up in a shell that doesn't include the pepper executable.

Also when I add extra-libraries: jq, pepper to cabal I keep getting this error:

Setup: Missing dependency on a foreign library:
  * Missing C library: pepper

It is a bit confusing because pepper is just a python executable, not a C library.

Thanks for your help.

Failing: nix-build --attr project1 release2.nix

Hi,

I was following along on this page: https://github.com/Gabriel439/haskell-nix/tree/master/project1

The above nix-build command has failed.

Even nix-build --attr project1 release3.nix is failing due to process pkg.

I tried using different versions 1.4*, 1.3, taking cue from the other .nix styles. More dependency failures and infinite-recursion failures are occurring now.

I am on ubuntu (16.04), Nix(1.11.16), Failing on Nix(2.0) too.

Pasting below the log:

---

Setup: Encountered missing dependencies:
QuickCheck ==2.8.*, process >=1.0 && <1.5
builder for ‘/nix/store/nkgwmcc89hqa5qys16s2vk1xf82381f7-optparse-applicative-0.13.0.0.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/kalxz4kz5n541gvpqqpidpdj6majn5lj-turtle-1.3.2.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/c308453anpfgsq9ka87m4wvwmgbvnyan-project1-1.0.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/c308453anpfgsq9ka87m4wvwmgbvnyan-project1-1.0.0.drv’ failed

---

nix-shell fails when passed `-A env`

On the most recent version of Nix the nix-shell -A env release0.nix command fails with the error:

error: attribute ‘env’ in selection path ‘env’ not found

Version info: nix-shell --version

nix-shell (Nix) 1.11.11

If you omit the -A env it works as expected.

Suggest fetching nixpkgs with fetchFromGitHub

fetchgit is considerably slower than fetchFromGitHub. At work, we use the following:

fetchFromGitHub
        {
          owner = "NixOS";
          repo = "nixpkgs";
          rev = "5ba7f33e3a4f8ebee9944e5e7f092edf4cb57f3e";
          sha256 = "0hkvwwaym689mr3j1iwa8xls735rdmxhf2w5x29j9yfmyrn2kdqy";
        }

Revision of approach

Hi Gabriel,

I sort of hit a dead-end with nix yesterday and reverted to regular cabal without having nix install it.

First of, I was trying to install hakyll with cabal install, but this failed with:

* Missing (or bad) header file: zlib.h
* Missing C library: zlib
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

I am on mac, so I decided to search a bit for this, but zlib should be part of standard install, and I located the folders. So that could not be the issue.

Then I read some people reinstalled their environment, which I did.
This resulted in nix-env --install cabal2nix breaking.

I asked around on freenode in nixos-darwin, and apparently it broke recently.

I was suggested to use 18.03 channel either for everything or just for cabal2nix.
FInally, I just removed all signs of nix and used regular cabal installed via homebrew and it worked straight away.

Maybe the tutorial should cover and switch to stable channels in the very beginning?

Question: add a link to haskell-overridez?

After reading Advanced Dependency Management and using it's ideas in some projects, I wrote haskell-overridez to make it really easy to use this approach.

Feedback on the project would be great; perhaps I could also send a PR adding a link to it on Advanced Dependency Management ?

Styx

I thought to mention the tool that I made to ease cabal/nix integration.

https://github.com/jyp/styx

Feel free to mention it in this documentation project, and conversely please let me know if you see glaring holes in it. Incidentally, I've made a fork of intero that you may be interested in, because it does not require "stack":

https://github.com/jyp/dante

Add example of initializing a new project from scratch

One thing that seems to be missing is a method to initialize a project from an empty directory.
So far, we have project folders with *.nix files in them already. There's no instruction about the best way to create a new project from scratch.

For example, we might suggest the following

Open a nix shell with ghc and cabal. Give the command for this.
Initialize the cabal info. Cabal init command
Generate the *.nix files. Probably use cabal2nix, and write the release.nix file

Prefer composable approach

The examples use methods that cause issues when trying to extend the package set further. Using the right tools, which are no harder, we can avoid frustration down the line.

• Use overlays instead of overrides. They are the way to go, because they are more powerful and, I'd argue, more consistent: overlays for packages sets, overrides for packages.
• Avoid rec keyword. People get used to it and will use it in overlay definitions, leading to surprises.
• Use haskellPackages.extend instead of .override { overrides =. It lets you add more layers of overrides which is more flexible. Attempting the same on the current code replaces the single layer of overrides with the new one, which is unexpected. As a bonus, it is syntactically simpler.

Nix works under WSL

Both Nix and GHC had problems under WSL (e.g. microsoft/WSL#743) but they've been fixed for a long time. I've played with Nix under WSL and it seems to be a very viable way of writing Haskell on Windows.

I think we should either remove the Nix/Windows disadvantage or qualify it some more.

Project 1 is not working anymore ?

I am trying to follow the examples of project 1 but release1.nix to release4.nix, are all failing with:

→ nix-build --attr project1 release4.nix                                                                                                                                                                                                   
error: attribute ‘project1’ missing, at /home/vagrant/projects/hub/haskell-nix/project1/release4.nix:22:16

release0.nix is ok and release5.nix is ok.

Maybe a recent change (I am using the latest stable channel of 17.09) has deprecated the exposed usage in favor of overlays or something alike ?

Recommend use of haskellLib.packagesFromDirectory

This function seems to do the same work as some of the code here: https://github.com/Gabriel439/haskell-nix/blob/master/project4/README.md. Is there overlap that could be reduced here?

Question: missing `exec` in a docker build with `dockerTools.buildImage`

When running a docker using this configuration I have got this error:

→ docker run -v ~/.config/cicd:/.config/cicd:ro pi3r/cicd-shell prod facts            
cicd: /bin/sh: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

I am still poking around the internet but I haven't found what I am missing yet. Any idea ?

Suggest upstreaming default.nix

You could possibly add a third option for turtle, where the user fetches from github, and then calls default.nix from the local clone, with the user upstreaming the default.nix to the turtle project itself.

Referencing source dependencies lead to shadowed dependency error

According to https://github.com/Gabriel439/haskell-nix/blob/master/project1/README.md#source-dependencies it's possible to reference a local source dependency via release.nix.

However, sometimes it leads to dependency shadowing (if I'm getting this error right):

libb >> nix-shell --command zsh
these derivations will be built:
  /nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv
  /nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv
building '/nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/nv4h00ma4lmwj9bnc8dhf9nzyj28d6ds-ghc-8.4.3.
ignoring (possibly broken) abi-depends field for packages
ignoring (possibly broken) abi-depends field for packages
unpacking sources
unpacking source archive /nix/store/nf0yf8xsi7l92ark5q08ddhrkh2syy9b-hi
source root is hi
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-liba-0.1.0.0.drv-0/setup-package.conf.d -j1 -threaded
Loaded package environment from /private/tmp/nix-build-liba-0.1.0.0.drv-0/hi/.ghc.environment.x86_64-darwin-8.4.3
<command line>: cannot satisfy -package-id liba-0.1.0.0-inplace:
    liba-0.1.0.0-inplace is unusable due to missing dependencies:
      aeson-1.2.4.0-KPiiXcK6dELrtdAOauWhV
    (use -v for more information)
builder for '/nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv' failed with exit code 1
cannot build derivation '/nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv' failed

So, there is libb depends on liba, and in libb/release.nix there is a reference:

{ compiler }:

let
  config = {
    packageOverrides = pkgs: rec {
      haskell = pkgs.haskell // {
        packages = pkgs.haskell.packages // {
          "${compiler}" = pkgs.haskell.packages."${compiler}".override {
            overrides = haskellPackagesNew: haskellPackagesOld: rec {
              liba = haskellPackagesNew.callPackage ../default.nix { };
              libb = haskellPackagesNew.callPackage ./default.nix { };
            };
          };
        };
      };
    };
  };

  pkgs = import <nixpkgs> { inherit config; };

in
  { project = pkgs.haskell.packages.${compiler}.libb;
  }

Minimal reproducible example could be found there - dmalikov/repro-shadowed-deps@7aa926d

Missing dependencies that fiailed when using GHCJS

Hi, I run nix-build based on ghcjs' compiler. Which is based on nixpkgs, it uses ghc7103. But when I build, i got the following error:

Configuring tasty-ant-xml-1.0.4...
Setup: At least the following dependencies are missing:
directory >=1.2.6.2, filepath >=1.4.1.0
builder for ‘/nix/store/wv8ynf0cpfn8b72y5lqa2bzyn50g9vgy-tasty-ant-xml-1.0.4.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/k9czcd3m6992n87gnza7wphwcqqgibfv-scientific-0.3.4.10.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/i830s057add78giqm80prjndissmwkvp-tasty-hunit-0.9.2’
cannot build derivation ‘/nix/store/lf2285k67bbc97bv6mk4kh32mdk4rj93-aeson-1.0.2.1.drv’: 1 dependencies couldn't be built

i've insert the missing dependencies in my release.nix in my previous attempt (this is because at first i'm using reflex, and now just plain ghcjs)

after searching around, i found this issue : NixOS/nixpkgs#22008.

THis is the same as my first problem ini here : NixOS/nixpkgs#23915

So in this situation, how we manage manage the environment of our compiler? I've add another built but the result as same.

Overlay example

Hey,

this is my first try using overlays, according to #53 this is the recommended way now.

It looks like nix is not able to find my app package,

nix-build release.nix 
error: attribute 'app' missing, at /Users/rsoeldner/work/haskell-nix-overlay/release.nix:11:12
(use '--show-trace' to show detailed location information)

do I still need to override haskell.packages ?
The default.nix was generated by cabal2nix . > default.nix

{ compiler ? "ghc802" }:

let
  a = self: super:
      {   
        app = super.callPackage ./default.nix;
      };  

  overlays = [a];
  pkgs = import <nixpkgs> { inherit overlays; };
in { app = pkgs.haskell.packages.${compiler}.app; }

nix-channel list gives no output

Hi.

I am on macOS running with nix 2.0.
My nix-channel --list gives not output.
What can be the cause of this? Does it just use the unstable without outputting?

After manually adding the unstable pkgs, the output is correct.

Error: file 'nixpkgs' was not found in the Nix search path

Hi,

After I ran nix-build release0.nix, it shows error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I), at /Users/allen/haskell-nix/project0/release0.nix:2:17

What should I do next? Thank you.

Passing `config` explicitly to nixpkgs imports ignores ~/.nixpkgs/config.nix

This might actually be intended, but there's a subtle different between the following two expressions:

import <nixpkgs> {}

and

import <nixpkgs> { config = ... }

The first form won't use config = {}. Instead, it loads $NIXPKGS_CONFIG or ~/.nixpkgs/config.nix and uses that for the config, while the second form ignores that file (providing no way for the user to customize the nixpkgs package set without checking the changes into the repo). I'm not sure if you want that, so feel free to close this issue if that is intended.

Shouldn't the docker image have its "Cmd" set to something non-null?

I'm probably just missing something, but I expected the last command to work.

project3 master % nix-build -A docker-container-small release2.nix
    /nix/store/s86imza199bw8dc9lv648w8zzf3cfp1z-docker-image-project3-container.tar.gz

project3 master % ls -l
    total 68K
    drwxr-xr-x 2 4.0K Jun 13 23:52 cbits
    -rw-r--r-- 1  348 Jun 13 23:52 default.nix
    -rw-r--r-- 1 1.5K Jun 13 23:52 LICENSE
    -rw-r--r-- 1   82 Jun 13 23:52 Main.hs
    -rw-r--r-- 1  520 Jun 13 23:52 project3.cabal
    -rw-r--r-- 1  110 Jun 13 23:52 Project3.hs
    -rw-r--r-- 1  21K Jun 13 23:52 README.md
    -rw-r--r-- 1  436 Jun 13 23:52 release0.nix
    -rw-r--r-- 1  630 Jun 13 23:52 release1.nix
    -rw-r--r-- 1 1.9K Jun 13 23:52 release2.nix
    lrwxrwxrwx 1   82 Jun 17 16:07 result -> /nix/store/s86imza199bw8dc9lv648w8zzf3cfp1z-docker-image-project3-container.tar.gz

project3 master % docker import result nix-haskell-test:latest
    sha256:7e57bf97c434974b0219d0dec5b2208319d666cbba07f1ae382f03d58bfa3e81

project3 master % docker inspect nix-haskell-test:latest | grep -iE '(id|created|cmd|zn)'
    "Id": "sha256:7e57bf97c434974b0219d0dec5b2208319d666cbba07f1ae382f03d58bfa3e81",
    "Created": "2017-06-17T21:16:27.785807674Z",
        "Cmd": null,
        "Cmd": null,
            "DeviceId": "4260",

project3 master % docker run nix-haskell-test:latest
    docker: Error response from daemon: No command specified.
    See 'docker run --help'.

Nothing from project4 builds correctly

For example:

$ nix-build release0.nix
...
Configuring optparse-applicative-0.13.0.0...
CallStack (from HasCallStack):
  die', called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:948:20 in Cabal-2.0.1.0:Distribution.Simple.Configure
  configureFinalizedPackage, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:470:12 in Cabal-2.0.1.0:Distribution.Simple.Con
figure
  configure, called at libraries/Cabal/Cabal/Distribution/Simple.hs:570:20 in Cabal-2.0.1.0:Distribution.Simple
  confHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:67:5 in Cabal-2.0.1.0:Distribution.Simple.UserHooks
  configureAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:174:19 in Cabal-2.0.1.0:Distribution.Simple
  defaultMainHelper, called at libraries/Cabal/Cabal/Distribution/Simple.hs:119:27 in Cabal-2.0.1.0:Distribution.Simple
  defaultMain, called at Setup.hs:2:8 in main:Main
Setup: Encountered missing dependencies:
process >=1.0 && <1.5
builder for '/nix/store/lml5p667kaialx856knj14yx7xr4zs8b-optparse-applicative-0.13.0.0.drv' failed with exit code 1
cannot build derivation '/nix/store/h2xlpmmd6ffbfz6d3sdinbfjyq2ax4sy-tasty-1.0.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/sanz1lf30sa950hacfada0f8raw9gshq-turtle-1.3.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n5mz1g9dq5qywgndicdgmapvs6hcf6rl-project4-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/n5mz1g9dq5qywgndicdgmapvs6hcf6rl-project4-1.0.0.drv' failed

project0: README.md: Text talks about LICENSE file being placed, expiriencing different result

https://github.com/Gabriel439/haskell-nix/blob/774325aeeba379f42d3e11ab824cc8c78876eb40/project0/README.md#L40-L58

Real result right now:

$ tree result/
result/
└── bin
    └── project0

1 directory, 1 file

I remember remotely there was work around distribution of LICENSE files.

Testing

Of course:

$ nixos-version
19.03pre153701.46651b82b87 (Koi)

Looked, and seems like it should be any other distro.

host$ docker run -it nixos/nix /bin/sh    # Alpine Linux
--- # Than you can literally run this:
nix-env -i git ghc tree
git clone https://github.com/Gabriel439/haskell-nix.git
cd haskell-nix/project0/
nix-build release0.nix
tree result
--- # And get the same result, now on non-NixOS Linux
result/
└── bin
    └── project0

1 directory, 1 file

Recommend using callHackage

Reasons:

• no need to keep cabal and generated expressions in sync
• NixOS/cabal2nix#217

Show how to incorporate ghcWithHoogle into shell.nix

Show how to incorporate ghcWithHoogle into shell.nix preferably without modifying default.nix that is generated with cabal2nix . > default.nix.

Use nix-shell --run rather than nix-shell --command

From the man page:

--command cmd
In the environment of the derivation, run the shell command cmd. This command is executed in an interactive shell. (Use --run to use a non-interactive shell instead.) However, a call to exit is implicitly added to the command, so the shell will exit after running the command. To prevent this, add return at the end; e.g. --command "echo Hello; return" will print Hello and then drop you into the interactive shell. This can be useful for doing any additional initialisation.

--run cmd
Like --command, but executes the command in a non-interactive shell. This means (among other things) that if you hit Ctrl-C while the command is running, the shell exits.

From the usage of --command in the tutorial (to run cabal configure), I think --run is really what you want.

Building project1 with cabal fails with dependency error

When trying to build project1 using the "Building with cabal" instructions from the project0 readme I get an error about it being unable to resolve dependencies.

Steps to reproduce

$ git clone https://github.com/Gabriel439/haskell-nix.git
$ cd haskell-nix/project1/
$ nix-build release0.nix
<snip:1>
$ ./result/bin/project1
Hello, world!
# Success! Works with nix-build.
$ nix-shell -A project1 release0.nix
$ cabal configure
Resolving dependencies...
Warning: solver failed to find a solution:
Could not resolve dependencies:
trying: project1-1.0.0 (user goal)
next goal: turtle (dependency of project1-1.0.0)
Dependency tree exhaustively searched.
Trying configure anyway.
Configuring project1-1.0.0...
cabal: Encountered missing dependencies:
turtle -any

So nix-build is able to resolve the turtle dependency but cabal configure is not. The steps above work when building and running project0 though.

Shell Output

snip:1 Output from nix-build release0.nix (irrelevant lines removed)
Full output at https://gist.github.com/m-renaud/b497478c059a79112393aeb015739730

these derivations will be built:
  /nix/store/lxk22wmjmwf2xadrw7pm37k80m9lcnql-project1-1.0.0.drv
<dropped>
Configuring project1-1.0.0...
Dependency base <5: using base-4.9.1.0
Dependency turtle -any: using turtle-1.3.6
Using Cabal-1.24.2.0 compiled by ghc-8.0
Using compiler: ghc-8.0.2
<dropped>
Installing executable(s) in
/nix/store/81gy6a8m1iin193507qyr580i2mj6n77-project1-1.0.0/bin
<dropped>

Versions

$ nix-build --version
nix-build (Nix) 1.11.11
$ nix-shell -A project1 release0.nix --run "cabal --version"
cabal-install version 1.24.0.2
compiled using version 1.24.2.0 of the Cabal library 
$ nix-shell -A project1 release0.nix --run "ghc --version"
The Glorious Glasgow Haskell Compilation System, version 8.0.2

Note about `verbosity` improvement

Verbosity

Nix derivations for Haskell projects are significantly more complex than their corresponding stack.yaml files. The release.nix files in this repository are the Nix analog of a stack.yaml file and you can see for yourself the increase in complexity as the examples progress in difficulty.

Nix Haskell users might be able to tackle this problem a bit in the coming months, by introducing more helper Nix functions...

Detail stack vs cabal vs nix cabal defaults

When invoking cabal, nix selects default behavior for cabal that might not match what someone would expect when coming from stack or cabal w/o nix. Consider spelling out main differences.

Mention disableSharedExecutable

This is a reminder to myself to mention the pkgs.haskell.lib.disableSharedExecutable trick for speeding up the load times of Haskell executables

Carving up into libraries for build time?

I am about to experiment with breaking up a large application into libraries to build fewer modules when rebuilding. In particular, I am interested in approaches which speed up builds between developers on a team, when there are small changes to modules in the middle or top layers of a large network of modules.

If you have made similar modifications to projects you work on, consider adding a section discussing tradeoffs.

Rename `default.nix` to `<project>.nix`

As I'm trying to make sense of Nix, I was slightly confused that I couldn't just nix-build the default.nix. https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure suggests another approach, where the generated nix expression is instead saved as <project>.nix and then called from default.nix and probably release*.nix by extension.

Talk about how to use GHC-specific tools (e.g. ghc-mod)

Tools like ghc-mod, as far as I understand, need to be built with the same version of GHC as the project you intend to use it in. Curious how you handle it with the setup described in this tutorial.

From Stack to Nix and beyond

Hey,
I'm still moving from Stack to Nix & cabal and this tutorial helped me a lot!
But a few general points aren't clear , at least for me.

• Do you suggest adding each dependency a separated file, for example like:
  $ cabal2nix cabal://turtle-1.3.2 > turtle.nix and add them to your project with the readDir trick ?

• Do you work inside the nix-shell ? Right now I'm moving from Stack and nix pick quite old packages and I'm not able to jump into the shell because of compile errors.

• How do you browse documentation ?

• How do you manage conflicts and dependencies between several packages.

Thank you very much for your tutorial 😄

Did you mean to repeat yourself in this line?

https://github.com/Gabriel439/haskell-nix/blame/a5214b16c00db1e9bf12134b92aca528eff3ff4f/project4/README.md#L262

dealing with Cabal "data-files" when minimizing closures

When minimizing closures, dependencies on projects like ekg that use Cabal "data-files" will have resultant binaries with baked-in references to the shared library of ekg in /nix/store. This makes the transitive closure bloat out again -- even when following the practice of statically linking and copying the final executable into a standalone derivation.

So I ended up doing the hack in this gist: https://gist.github.com/shajra/afed560778a556e487b201b5b4c2ad52

It's definitely a hack, but it works. I tried to figure out a less hacky way to do this with Cabal, but made no real progress.

This is one of those things that's half Haskell, half Nix, so it's not always clear which community to go to first for help. This project seems like a way to get some intersectional conversation.

I'll leave it to @Gabriel439's discretion of how/whether to adjust this project. I don't know if there's a point where advanced topics are more of a distraction than help. Mostly, I wanted feedback. And if we reach a consensus, we can see how it affects the guide or not.