railwayapp/nixpacks

App source + Nix packages + Docker = Image

nixpacks's Issues

Read Node version from more locations

• .nvmrc file
• .node-version file
• NODE_VERSION environment variable

RFC: Provider Caching

Providers should be able to specify what files are necessary for the install step to allow for better Docker layer caching.

For example, the NPM provider could specify that package.json is all that is necessary to install. If that files does not change between deploys, a cached layer can be used.

Provider caching can be split up into a few different types

1. Specify the files necessary to install so that only these files are keyed against when computing a cache hit/miss
2. App/framework level caches (e.g. .next directory). These caches should be re-used even if the source code that generated the cache changes. For this level of caching we cannot take advantage of built in Docker layer caching and will likely require a persistent volume that can be shared between builds.

This RFC will mainly focus on the first type.

Staticfile/HTML provider

Yarn 2 support

Haskell Cabal provider

Improve Procfile parsing

The Procfile parsing is very basic at the moment. We should look for and use other process types besides "web" if web is not found. If more than a single process type is found, we should also warn/error to provide feedback to the user.

`.rust-toolchain` support

support installing different rust toolchains

OCaml provider

Support `deno.json` and import maps in Deno

Deno has deno.json, which would be a pretty foolproof way to detect it (along with looking for TS/JS files). Also, Deno now has import maps, which allow for bare module specifiers, which means that the regex currently used for detection (which relies on the app importing modules from https://deno.land/ via absolute URLs) won't necessarily work for all Deno apps.

C# Provider

Part of #84

Lua provider

Elixir provider

Env Var Config

Allow different parts of the build to be configurable by environment variables.

For example

• Build command
• Nix packages to install

RFC: Support other container runtimes

I specifically have in mind Podman.

This could probably just be done as a fallback - if Docker's found, use it, else try Podman, etc.

Also, Podman has some interesting features that might be useful for Nixpacks - for instance, Buildah, which is a tool for building containers imperatively.

Haskell provider doesn't work on M1 macs

Running Nixpackgs on the examples/haskell-stack directory fails with error

Unable to find installation URLs for OS key: linux-aarch64-tinfo6

It seems that the Haskell toolchain cannot find binaries for the aarch64 architecture on Linux

python-poetry provider

Vlang provider

https://vlang.io

Haskell provider

Test Issue Please Ignore

Dart provider

Clojure provider

error ```node:internal/modules/cjs/loader:1183```

i get this error if i run my code and the deploy is active after that he crash

node:internal/modules/cjs/loader:1183
return process.dlopen(module, path.toNamespacedPath(filename));
^
Error: libuuid.so.1: cannot open shared object file: No such file or directory
at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:999:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/app/node_modules/canvas/lib/bindings.js:3:18)
at Module._compile (node:internal/modules/cjs/loader:1099:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
code: 'ERR_DLOPEN_FAILED'
}
Node.js v17.9.0

Glob failure on Windows

The glob crate contains an error which prevents it from functioning on Windows style paths.
This causes the following tests to fail:

• nixpacks::app::tests::test_find_files
• nixpacks::app::tests::test_find_match
• nixpacks::app::tests::test_strip_source_path

Ruby provider

C provider

Ruby on Rails provider

• Provider for Ruby
• Should support Sinatra and Rails

Python provider

If package manifest isn't found, don't do a clean install

This is causing issues because some people don't commit yarn.lock & package-lock.json. We need to remove --frozen-lockfile in Yarn if yarn.lock doesn't exist, and we need to do npm i instead of npm ci if package-lock.json doesn't exist.

Zig provider

https://ziglang.org/

Tset

Nixpacks base debian image doesn't work on M1 macs

Building with the base debian image hosted on GHCR doesn't work on M1 macs and fails with the following error. Assuming it is because Nix installed on x86 hardware but is installing packages on arm64.

Building the base image locally and referencing that works as expected.

#8 18.69 copying path '/nix/store/7b2z05mgfwgb58ya4i3gynhvjw22v29q-xz-5.2.5-bin' from 'https://cache.nixos.org'...
#8 18.74 copying path '/nix/store/j72mw6h364nmk7l3mq6gqwjd568l7r8l-perl-5.34.1' from 'https://cache.nixos.org'...
#8 24.70 copying path '/nix/store/hfcv57qibyrqha8gmnk0mkz6k9z3krhy-sqlite-3.38.3' from 'https://cache.nixos.org'...
#8 24.95 copying path '/nix/store/lq3ljqhz6zwkwsyvjyw7750n35psz4fs-openssl-1.1.1o-bin' from 'https://cache.nixos.org'...
#8 25.14 copying path '/nix/store/01kia41csjia67pry1rv828i9pvnnqfq-python3-3.9.12' from 'https://cache.nixos.org'...
#8 36.30 copying path '/nix/store/v3gg98k5gd0fjfx1wf6vvjmj2idk1gg6-openssl-1.1.1o-dev' from 'https://cache.nixos.org'...
#8 36.47 copying path '/nix/store/g146gm78fviybvpdipvmgchv5ikv9d4h-stdenv-linux' from 'https://cache.nixos.org'...
#8 36.49 copying path '/nix/store/sm20x1x5mly3r9adv751ipczbq8p66a5-zlib-1.2.12-dev' from 'https://cache.nixos.org'...
#8 36.53 copying path '/nix/store/vyyar1czkx02agclv9z60mbsmh54l5aq-nodejs-16.15.0' from 'https://cache.nixos.org'...
#8 42.88 error: unable to load seccomp BPF program: Invalid argument
#8 42.88 (use '--show-trace' to show detailed location information)
------
executor failed running [/bin/bash -o pipefail -c nix-env -if environment.nix]: exit code: 1
Error: Docker build failed

The workaround is fine but ideally nixpacks works on all architectures. I think it might make sense to revert back to the old way of setting the base image.

Thoughts @wyzlle

Deno provider

.NET Core provider

Php provider

Environment variable support

Much like docker and pack, you should be able to provide environment variables that will be available to the build.

I am thinking they can be supplied with the flag

--env "NAME=value" EXTERNAL

In the NAME=value case, the variable NAME is set to "value". In the second case, the variable EXTERNAL is provided from the calling environment.

Scala Provider

problem building a nixpack under gitpod

I am using gitpod as my dev environment as I learn about nixpacks. I can build the binary without any problem but I can't build a nitpack for the example node project. I'm not very familiar with rust so not really clear what the error message is indicating.

Note, I am able to build locally on my Mac so its something to do with the gitpod environment. It looks like it fails doing a npm ci at the end of the docker build. My Mac has node 17 while gitpod is 16. Not sure if that's important.

To reproduce the issue, create a gitpod workspace using https://gitpod.io/# + the GitHub repo URL. Then from the terminal run cargo run -- build examples/node --name node

Here is the output.

cargo run -- build examples/node --name node
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/nixpacks build examples/node --name node`
=== Building (nixpacks v0.0.8) ===
=> Generated new build plan
=> Packages
    -> nodejs
=> Install
    -> npm ci
=> Build
    -> Skipping
=> Start
    -> npm run start
=> Building image with Docker
Sending build context to Docker daemon  7.168kB
Step 1/15 : FROM debian:bullseye-slim
 ---> bfbec70f8488
Step 2/15 : RUN apt-get update && apt-get -y upgrade   && apt-get install --no-install-recommends -y locales curl xz-utils ca-certificates openssl   && apt-get clean && rm -rf /var/lib/apt/lists/*   && mkdir -m 0755 /nix && mkdir -m 0755 /etc/nix && groupadd -r nixbld && chown root /nix   && echo 'sandbox = false' > /etc/nix/nix.conf   && for n in $(seq 1 10); do useradd -c "Nix build user $n" -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(command -v nologin)" "nixbld$n"; done
 ---> Using cache
 ---> 6b66b20f974c
Step 3/15 : SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 ---> Using cache
 ---> c550359f3833
Step 4/15 : RUN set -o pipefail && curl -L https://nixos.org/nix/install | bash     && /nix/var/nix/profiles/default/bin/nix-collect-garbage --delete-old
 ---> Using cache
 ---> 8cd081977e35
Step 5/15 : ENV   ENV=/etc/profile   USER=root   PATH=/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin   GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt   NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt   NIX_PATH=/nix/var/nix/profiles/per-user/root/channels
 ---> Using cache
 ---> 7e8e1d37c7e4
Step 6/15 : RUN nix-channel --update
 ---> Using cache
 ---> 9fcce4c4d9bd
Step 7/15 : RUN mkdir /app/
 ---> Using cache
 ---> da6c338bda53
Step 8/15 : WORKDIR /app/
 ---> Using cache
 ---> ddab151a4376
Step 9/15 : COPY environment.nix /app/
 ---> Using cache
 ---> d4e0409d43c1
Step 10/15 : RUN nix-env -if environment.nix
 ---> Using cache
 ---> 361901493d55
Step 11/15 : ARG NPM_CONFIG_PRODUCTION NODE_ENV
 ---> Running in 2ac92b375b9b
Removing intermediate container 2ac92b375b9b
 ---> f97b63c6066e
Step 12/15 : ENV NPM_CONFIG_PRODUCTION=$NPM_CONFIG_PRODUCTION NODE_ENV=$NODE_ENV
 ---> Running in 651966ee5147
Removing intermediate container 651966ee5147
 ---> 518edade8cc9
Step 13/15 : COPY . /app/
 ---> 8205e15c5e95
Step 14/15 : RUN npm ci
 ---> Running in d1a655c5b9f7
/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node[1]: ../src/node_platform.cc:61:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xa66fa8 node::Abort() [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 2: 0xa67037  [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 3: 0xae82a5 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 4: 0xae83d2 node::NodePlatform::NodePlatform(int, v8::TracingController*) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 5: 0xa270ff node::V8Platform::Initialize(int) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 6: 0xa2541b node::InitializeOncePerProcess(int, char**, node::InitializationSettingsFlags, node::ProcessFlags::Flags) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 7: 0xa255a9 node::InitializeOncePerProcess(int, char**) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 8: 0xa25608 node::Start(int, char**) [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
 9: 0x7f6b1de611d7  [/nix/store/m4g6lswi75b739cpdx8wfxlfmcazyks9-glibc-2.34-115/lib/libc.so.6]
10: 0x7f6b1de61297 __libc_start_main [/nix/store/m4g6lswi75b739cpdx8wfxlfmcazyks9-glibc-2.34-115/lib/libc.so.6]
11: 0x98be61 _start [/nix/store/7fy3g0bbiww61k7imi4bayfnjxmbs3cj-nodejs-16.15.0/bin/node]
The command '/bin/bash -o pipefail -c npm ci' returned a non-zero code: 139
Error: Docker build failed

Golang integration tests

RFC: Managing System Resources Using Nixpacks

Managing system resources (RAM, stack size, max file watch, CPU time, etc.) using Nixpacks configuration would be a helpful feature. As of now one will have to use custom commands to do that. Making this a core feature will allow users to set limits per phase easily. This can be done by using tools like prlimit, ulimit and timeout or by creating our own wrapper. Also, this is how Piston (remote code execution engine) sets resource limits.

Use regular caching on CI

nixpacks/.github/workflows/ci.yml

Line 46 in eb1c51c

- uses: satackey/[email protected]

Is there any reason we are using this instead of just @actions/cache or some Rust configured one? Or is this doing something else?

Crystal provider

https://crystal-lang.org/

RFC: Build Plan Format

As more and more features have been added to Nixpacks, the build plan has slowly evolved. Before it gets out of control, we should settle on a finalized build plan format that is human readable and intuitive.

The current build plan can be seen in the following screenshot

I propose we flatten and simplify the format.

interface NixPackage {
  name: string;
  overlay?: string;
}

interface SetupPhase {
  nixPkgs: NixPackage[];
  onlyIncludeFiles?: string[];
}

interface InstallPhase {
  cmd?: string;
  onlyIncludeFiles?: string[];
}

interface BuildPhase {
  cmd?: string;
  onlyIncludeFiles?: string[];
}

interface StartPhase {
  cmd?: string;
}

interface BuildPlan {
  setup?: SetupPhase;
  install?: InstallPhase;
  build?: BuildPhase;
  start: StartPhase;
  variables: Record<string, string>;
}

Example: Yarn fastify

{
  "setup": {
    "nixPkgs": [{ "name": "pkgs.stdenv" }, { "name": "pkgs.yarn" }]
  },
  "install": {
    "cmd": "yarn install --frozen-lockfile",
    "onlyIncludeFiles": ["package.json", "yarn.lock"]
  },
  "build": {
    "cmd": "yarn build"
  },
  "start": {
    "cmd": "yarn start"
  },
  "variables": {
    "NPM_CONFIG_PRODUCTION": "false",
    "NODE_ENV": "production"
  }
}

Example: Rust rocket

{
  "setup": {
    "nixPkgs": [
      { "name": "gcc" },
      {
        "name": "(rust-bin.fromRustupToolchainFile ./rust-toolchain.toml)",
        "overlay": "https://github.com/oxalica/rust-overlay/archive/master.tar.gz"
      }
    ],
  "onlyIncludeFiles": ["rust-toolchain.toml"]
  },
  "build": {
    "dependsOn": "setup",
    "cmd": "cargo build --release",
    "onlyIncludeFiles": ["Cargo.toml"]
  },
  "start": {
    "cmd": "./target/release/rocket",
    "image": "alpine"
  },
  "variables": {
    "ROCKET_ADDRESS": "0.0.0.0"
  }
}

PNPM provider

NodeJS version parsing

We should parse the expected Node version from common locations such as package.json engines and .nvmrc.

We can find the exact version on nixpkgs using something like https://lazamar.co.uk/nix-versions/. However, I think initially just supporting the latest major versions is a good middle ground and will handle 90+% of use cases and will allow us to use the latest version of nixpkgs.

Java provider

PureScript provider

https://www.purescript.org/

RFC: File based config

Nixpacks configuration should be able to be provided through a file on disk in the app source. The main benefit is that build configuration can be tracked through source control.

At the moment, the configurable options are

• Nixpkgs to install
• Build command
• Start command

A very basic format for these options is a JSON file

{
  "packages": ["pkgs.cowsay", "pkgs.lolcat"],
  "startCommand": "...",
  "buildCommand": "..."
}

Support `rust-version`

You can use the rust-version field on package in Cargo.toml: (docs)

`build` fails on Windows, due to `cp` not existing

The cp shell command is not available on Windows, therefore building fails when the program attempts to copy source files into a temporary directory.

C++ provider