samrose/setup.nix

Nixpkgs based build tools for declarative Python packages

setup.nix – Nix for Python developers simplified

setup.nix provides opinionated helper functions and pip2nix-based workflow for developing, testing and packaging declaratively configured Python packages in Nix/NixOS environments. setup.nix is designed for mixed environments, where both traditional and Nixpkgs based Python package development must coexist with minimal additional maintanance.

setup.nix does not replace any tools or conventions in Nixpkgs, but helps to develop Python packages on top of it when not all required packages or versions are yet (or no longer) in Nixpkgs.

Note

The current master is development version of setup.nix 2.0 supporting NixOS 18.09, pip >= 18 and implicit reuse of nixpkgs Python package derivations.

Quick start

Requirements

• package configuration declaratively in setup.cfg
• union of package and development requirements in requirements.txt
• Nix or NixOS with current Nixpkgs channel.

Installation

Create minimal ./setup.nix:

{ pkgs ? import <nixpkgs> {}
, pythonPackages ? pkgs.python3Packages
, setup ? import (fetchTarball {
    url = "https://github.com/datakurre/setup.nix/archive/v.1.1.1.tar.gz";
    sha256 = "1lxw6ifsc1psrmaz0wrz2jymdsxkh3abrw938ch59dkf3g9z3bm4";
  })
}:

setup {
  inherit pkgs pythonPackages;
  src = ./.;
}

Generate requirements.nix from your requirements.txt:

$ nix-shell setup.nix -A pip2nix \
  --run "pip2nix generate -r requirements.txt --output=requirements.nix"

Basic use cases

Develop package in console with a Nix development shell (this is similar to developing with a regular Python virtualenv):

$ nix-shell setup.nix -A develop

Build easily accessible environment with all the requirements (this is useful e.g. as project Python interpreter for PyCharm):

$ nix-build setup.nix -A env

Build a reasonably minimal docker image from the package (the best part being that build itself does not requier Docker at all):

$ nix-build setup.nix -A bdist_docker
$ docker load < result

Install the package for local use (that's where Nix excels, because any amount of Python packages could be installed to be available in path without worrying about conflicting package versions):

$ nix-env -f setup.nix -iA build

Build a wheel release for the package (though sure you could just include zest.releaser [recommended] in your requirements.txt and use that):

$ nix-build setup.nix -A bdist_wheel

Integration with regular Makefile so that make nix-test will be equal to make test within Nix-built shell:

nix-%: requirements.nix
   nix-shell setup.nix -A develop --run "$(MAKE) $*"

Troubleshooting

When Python packages fail to build with nix-shell or nix-build, it's usually because of missing buildInputs (because pip2nix cannot detect setup_requires for generated packages in requirements.nix). These issues can usually be fixed by manually overriding package derivation in setup.nix overrides. Check the automatically included default overrides for reference.

Until all the available features and options are documented, see the setup-function and examples for more information.

Complete example

Here's a complete example of using setup.nix for Python package development:

Project skeleton

./helloworld.py:

# -*- coding: utf-8 -*-
def main():
    print('Hello World!')

./tests/test_helloworld.py:

# -*- coding: utf-8 -*-
import helloworld


def test_main():
    helloworld.main()

./setup.py:

from setuptools import setup; setup()

./setup.cfg:

[metadata]
name = helloworld
version = 1.0

[options]
setup_requires =
    pytest-runner
install_requires =
tests_require =
    pytest
py_modules =
    helloworld

[options.entry_points]
console_scripts =
    hello-world = helloworld:main

[aliases]
test = pytest

./requirements.txt:

coverage
pytest
pytest-cov
pytest-runner

./setup.nix:

{ pkgs ? import <nixpkgs> {}
, pythonPackages ? pkgs.python3Packages
, setup ? import (fetchTarball {
    url = "https://github.com/datakurre/setup.nix/archive/v.1.1.1.tar.gz";
    sha256 = "1lxw6ifsc1psrmaz0wrz2jymdsxkh3abrw938ch59dkf3g9z3bm4";
  })
}:

setup {
  inherit pkgs pythonPackages;
  src = ./.;
  doCheck = true;
  image_entrypoint = "/bin/hello-world";
}

./requirements.nix:

$ nix-shell setup.nix -A pip2nix \
    --run "pip2nix generate -r requirements.txt --output=requirements.nix"

./tests.nix:

{ pkgs, pythonPackages, make-test, build, ... }:

make-test ({ pkgs, ... }: {
  name = "test";
  machine = { config, pkgs, lib, ... }: {
    environment.systemPackages = [ build ];
  };
  testScript = ''
    $machine->waitForUnit("multi-user.target");
    $machine->succeed("hello-world") =~ /Hello World!/;
  '';
})

Interaction examples

Run tests with coverage:

$ nix-shell setup.nix -A develop --run "pytest --cov=helloworld"

Build and run docker image:

$ docker load < `nix-build setup.nix -A bdist_docker --no-build-output`
$ docker run --rm helloworld:latest
Hello World!

Run functional NixOS tests:

$ nix-build setup.nix -A tests

Configuration options

Here is the signature of setup.nix expression with all the available configuration arguments:

{ pkgs ? import <nixpkgs> {}
, pythonPackages ? pkgs.pythonPackages

# project path, usually ./., without cleanSource, which is added later
, src

# custom post install script
, postInstall ? ""

# enable tests on build
, doCheck ? false

# requirements overrides fix building packages with undetected inputs
, overrides ? self: super: {}
, defaultOverrides ? true
, implicitOverrides ? true

# force to build environments without package level dependency checks
, force ? false
, ignoreCollisions ? false

# non-Python inputs
, buildInputs ? []
, propagatedBuildInputs ? []
, shellHook ? ""

# known list of "broken" as in non-installable Python packages
, nonInstallablePackages ? [ "zc.recipe.egg" ]

# very dedicated bdist_docker
, image_author ? null
, image_name ? null
, image_tag ? "latest"
, image_entrypoint ? "/bin/sh"
, image_cmd ? null
, image_features ? [ "busybox" "tmpdir" ]
, image_labels ? {}
, image_extras ? []
, image_created ? "1970-01-01T00:00:01Z"
, image_user ? { name = "nobody"; uid = "65534"; gid = "65534"; }
, image_keepContentsDirlinks ? false
, image_runAsRoot ? ""
, image_extraCommands ? ""
, image_extraConfig ? {}
}:

Arguments in detail:

pkgs

setup.nix defaults to the currently available Nixpkgs version, but also accepts the given version for better reproducibility:

{
  pkgs= import (fetchTarball {
    url = "https://github.com/NixOS/nixpkgs-channels/archive/ef450efb9df5260e54503509d2fd638444668713.tar.gz";
    sha256 = "1k9f3n2pmdh7sap79c8nqpz7cjx9930fcpk27pvp6lwmr4qigmxg";
  }) {}
}

pythonPackges

In Nixpkgs each Python version has its own set of available packages. This is also used in setup.nix for selection of the used Python version (e.g. pkgs.python27Packages for Python 2.7 and pkgs.pythonPackages36Packages for Python 3.6).

src

This is the absolute path for the project directory or environment.nix. Usually this must be src = ./. in Nix for setup.nix to properly find your project's setup.cfg and requirements.txt. If you are only building an evironment or an existing package from requirements.txt, src = ./requirements.nix is enough.

force

By default setup.nix tries its best to behave like a good nixpkgs citizen and compose Python projects from reusable package builds with well-defined dependencies. force = true configures setup.nix to build individual packages without their dependencies, only to add all the dependencies into the final derivation. This makes it possible to build packages with circular dependencies or packages with add-ons (depending on the package itself).

doCheck

In Nixpkgs it is usual to require tests to pass before pakage is built, but elsewhere it's usual to run tests in a separate test stage on CI. setup.nix defaults to disable automatic tests on build, but tests can be forced with argment doCheck = true.

overrides

Because pip2nix cannot always generate fully working derivations for every Python package, overrides-function is required to complete the failing derivations. In addition, some Python package are actually hard to build, but luckily it's possible to re-use build insructions from Nixpkgs. See the default overrides example function (overrides = self: super: {}).

The most usual use cases for overrides are:

1. Adding missing Python buildInputs from package setup_requires or non-Python inputs required by possible C-extensions in the package.
2. Using the existing Nixpkgs derivation as it is.
3. Using use the existing Nixpkgs derivation with updated PyPI version.

defaultOverrides

setup.nix includes growing amount default package overrides to minimize the need of custom overrides. In case that those default overrides cause unexpected issues, it's possible to disable including the with argument defaultOverrides = false.

buildInputs

Non-Python build-time dependencies (usually Nixpkgs-packages) required for building or testing the developed Python package.

propagatedBuildInputs

Non-Python run-time dependencies (usually Nixpkgs-packages) required for actually using the developed Python package.

image_name, image_tag, image_entrypoint, image_features, image_labels:

Required for configuring the build of Docker image with bdist_docker build target.

Allowed arguments for image_features are:

• "busybox" to make possible to execute interactive shell in the image with e.g. docker run --rm -ti --entrypoint=/bin/sh
• "tmpfile" to include writable /tmp in the image with environment variables TMP and HOME set to point it.

image_labels should be a flat record of key value pairs for to be used as Docker image labels.

More examples

• https://github.com/collective/sphinxcontrib-httpexample
• https://github.com/datakurre/setup.nix/blob/master/examples/env
• https://github.com/datakurre/setup.nix/blob/master/examples/package
• https://github.com/datakurre/setup.nix/blob/master/examples/tool