target / lorri Goto Github PK
View Code? Open in Web Editor NEWYour project's nix-env
License: Apache License 2.0
Your project's nix-env
License: Apache License 2.0
When installing from e943fa4 I get this build error.
Running cargo build --release
error: failed to load source for a dependency on `directories`
Caused by:
Unable to update registry `https://github.com/rust-lang/crates.io-index`
Caused by:
failed to update replaced source registry `https://github.com/rust-lang/crates.io-index`
Caused by:
failed to parse manifest at `/build/lorri-vendor/fuchsia-cprng/Cargo.toml`
Caused by:
editions are unstable
Caused by:
feature `edition` is required
consider adding `cargo-features = ["edition"]` to the manifest
builder for '/nix/store/ah0x7qlcg0pzskk3vhq8cwfqji5vqkra-lorri.drv' failed with exit code 101
The current default.nix
file assumes that lorri is always installed from the project git directory. This causes major headaches when trying to use lorri with tools like niv or packaging it.
Maybe it would be possible to support an alternative "portable" default.nix
file?
We are a companion tool to direnv, so we should use the same conventions.
direnv status is the same as lorri info, so we should rename while we still can.
If you do a naive src = ./.
somewhere, nix will have to re-import the whole directory into the store each time any file in ./.
changes, which usually includes things like ./.git
or ./dist
or ./target
, meaning lorri will start many useless evaluations.
See #6 as an example.
You can use builtins.filterSource
and the functions in nixpkgs/lib/sources.nix
to filter which files nix imports (and thus which files lorri watches), but they are extremely unintuitive and notoriously hard to use and debug.
pkgs/build-support/nix-gitignore
or https://github.com/Profpatsch/nixperiments/blob/master/filterSourceGitignore.nix are another practical approach that uses the projects .gitignore
files.
So we basically need three (four) things:
builtins.filterSource
(#84)Related: #49
I'm currently on direnv-2.17.0
(and nix
2.1.3 on Darwin), and it looks like direnv/nix.nix
is installing my current version of direnv
instead of 2.19.2
.
It looks like isOlderThan
is determining that here:
(builtins.compareVersions wantedVersion pkgVersion) == 0
And testing that out gives me:
nix-repl> (builtins.compareVersions "2.19.2" "2.19.3")
-1
nix-repl> (builtins.compareVersions "2.19.2" "2.19.2")
0
nix-repl> (builtins.compareVersions "2.19.2" "2.17.0")
1
Should that line instead be the below?
(builtins.compareVersions wantedVersion pkgVersion) >= 0
Still very much a Nix beginner, though, so it's entirely possible I'm missing something!
I've got GOPATH setup from my default shell, and using direnv's default use_nix
my GOPATH is appended to in the resulting env, but with lorri it is replaced.
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ ls -l ~/.config/direnv/
total 20
drwxr-xr-x 2 manny users 34 Apr 2 14:25 allow
-rw-r--r-- 1 manny users 389 Apr 2 13:12 direnvrc.0
-rw-r--r-- 1 manny users 4017 Mar 19 11:28 direnvrc.old
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ direnv deny
direnv: error .envrc is blocked. Run `direnv allow` to approve its content.
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ cd ..
direnv: unloading
[manny@manny:~/code/src/github.com/packethost]$ cd -
/home/manny/code/src/github.com/packethost/tinkerbell
direnv: error .envrc is blocked. Run `direnv allow` to approve its content.
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ echo $GOPATH
/home/manny/code
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ echo 'use_nix' > .envrc
direnv: loading .envrc
direnv: export +AR +AR_FOR_TARGET +AS +AS_FOR_TARGET +CC +CC_FOR_TARGET +CONFIG_SHELL +CXX +CXX_FOR_TARGET +HOST_PATH +IN_NIX_SHELL +LD +LD_FOR_TARGET +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_aarch64_unknown_linux_gnu_TARGET_TARGET +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_TARGET +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_aarch64_unknown_linux_gnu_TARGET_TARGET +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_TARGET +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_STORE +NIX_TARGET_BINTOOLS +NIX_TARGET_CC +NIX_TARGET_CFLAGS_COMPILE +NIX_TARGET_LDFLAGS +NM +NM_FOR_TARGET +OBJCOPY +OBJCOPY_FOR_TARGET +OBJDUMP +OBJDUMP_FOR_TARGET +RANLIB +RANLIB_FOR_TARGET +READELF +READELF_FOR_TARGET +SIZE +SIZE_FOR_TARGET +SOURCE_DATE_EPOCH +STRINGS +STRINGS_FOR_TARGET +STRIP +STRIP_FOR_TARGET +TARGET_AR +TARGET_AS +TARGET_CC +TARGET_CXX +TARGET_LD +TARGET_NM +TARGET_OBJCOPY +TARGET_OBJDUMP +TARGET_RANLIB +TARGET_READELF +TARGET_SIZE +TARGET_STRINGS +TARGET_STRIP +TARGET_WINDRES +TEMP +TEMPDIR +TMP +WINDRES +WINDRES_FOR_TARGET +_PATH +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +env +name +nativeBuildInputs +out +outputs +propagatedBuildInputs +propagatedNativeBuildInputs +shell +stdenv +strictDeps +system ~GOPATH ~PATH
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ echo $GOPATH
/home/manny/code:/nix/store/l6747b8vkpmazcaxm89dd0amky0q35nn-go-1.11/share/go
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ echo 'eval "$(lorri direnv)"' > .envrc
direnv: error .envrc is blocked. Run `direnv allow` to approve its content.
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ direnv allow
direnv: loading .envrc
direnv: export +AR +AR_FOR_TARGET +AS +AS_FOR_TARGET +CC +CC_FOR_TARGET +CONFIG_SHELL +CXX +CXX_FOR_TARGET +HOST_PATH +IN_NIX_SHELL +LD +LD_FOR_TARGET +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_aarch64_unknown_linux_gnu_TARGET_TARGET +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_TARGET +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_aarch64_unknown_linux_gnu_TARGET_TARGET +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_TARGET +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_LOG_FD +NIX_STORE +NIX_TARGET_BINTOOLS +NIX_TARGET_CC +NIX_TARGET_CFLAGS_COMPILE +NIX_TARGET_LDFLAGS +NM +NM_FOR_TARGET +OBJCOPY +OBJCOPY_FOR_TARGET +OBJDUMP +OBJDUMP_FOR_TARGET +RANLIB +RANLIB_FOR_TARGET +READELF +READELF_FOR_TARGET +SIZE +SIZE_FOR_TARGET +SOURCE_DATE_EPOCH +STRINGS +STRINGS_FOR_TARGET +STRIP +STRIP_FOR_TARGET +TARGET_AR +TARGET_AS +TARGET_CC +TARGET_CXX +TARGET_LD +TARGET_NM +TARGET_OBJCOPY +TARGET_OBJDUMP +TARGET_RANLIB +TARGET_READELF +TARGET_SIZE +TARGET_STRINGS +TARGET_STRIP +TARGET_WINDRES +WINDRES +WINDRES_FOR_TARGET +_PATH +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +env +name +nativeBuildInputs +origArgs +origBuilder +origPATH +origSystem +out +outputs +propagatedBuildInputs +propagatedNativeBuildInputs +shell +stdenv +strictDeps +system ~GOPATH ~PATH
[manny@manny:~/code/src/github.com/packethost/tinkerbell]$ echo $GOPATH
/nix/store/l6747b8vkpmazcaxm89dd0amky0q35nn-go-1.11/share/go
I see that the generated output of lorri direnv
handles PATH
as a special variable, should the logic be updated for *PATH
or add GOPATH
to the list?
~> cd opt/lorri/example/
direnv: loading .envrc
/bin/bash: line 66: /Users/xxx/Library/Caches/com.github.target.lorri.lorri.lorri/example/gc_root/build-0: No such file or directory
direnv: error exit status 1
~/o/l/example> cd..
fish: Unknown command cd..
direnv: loading .envrc
/bin/bash: line 29: Fusion.app/Contents/Public:/Users/xx/.nix-profile/bin:/Users/xx/.nix-profile/lib/kde4/libexec:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default:/nix/var/nix/profiles/default/lib/kde4/libexec:/Applications/kitty.app/Contents/MacOS:/Users/xx/bin: No such file or directory
direnv: error exit status 1
I have a little postgresql state dir inside my project's dot files, and when PG is running, it does changes some files regularly
[2019-03-29T06:46:35Z DEBUG lorri::watch] Event path ("/home/danbst/dev/jobs/_build/pgdata/base/13393/2601") parent ("/home/danbst/dev/jobs/_build/pgdata/base/13393") matches watched path
[2019-03-29T06:46:35Z DEBUG lorri::watch] Watch Event: RawEvent {
path: Some(
"/home/danbst/dev/jobs/_build/pgdata/base/13393/2601"
),
op: Ok(
CLOSE_WRITE
),
cookie: None
}
[2019-03-29T06:46:35Z DEBUG lorri::watch] watch event: (
Ok(
CLOSE_WRITE
),
Some(
"/home/danbst/dev/jobs/_build/pgdata/base/13393/2601"
)
)
[2019-03-29T06:46:35Z DEBUG lorri::watch] Received event: RawEvent {
path: Some(
"/home/danbst/dev/jobs/_build/pgdata/base/13393/2601"
),
op: Ok(
CLOSE_WRITE
),
cookie: None
}
[2019-03-29T06:46:35Z INFO lorri::watch] Found 28 events
Started
[2019-03-29T06:46:36Z DEBUG lorri::build_loop] original paths: 1235
[2019-03-29T06:46:36Z DEBUG lorri::build_loop] -> reduced to: 4
[2019-03-29T06:46:36Z DEBUG lorri::build_loop] named drvs: {
"shell_gc_root": "/nix/store/dhziglbpkwx0ssmq21lckcwc394wb623-lorri-keep-env-hack-shell.drv",
"shell": "/nix/store/4w3nl3h4x0gl3q6myf6x0nl6hy8ldfsj-shell.drv"
}
It causes regular direnv reloads, like every 10 seconds and direnv debug log on each trivial command.
Is there way to exclude files from watching, probably even use .gitignore
?
While editing a shell.nix
being monitored by lorri, I accidentally wrote import ./default { }
😇 instead of import ./. {}
or import ./default.nix {}
-- which of course causes a Nix eval error but lorri seems to die trying to deal with the path that doesn't exist.
Lorri just was trying to watch things for me and in return I cause it to panic and die :(. Poor lorri. Sokay these guys can fix you up, they're the best! ;)
Shea Profpatsch and I met a couple weeks ago and came up with some designs on how to setup a daemon.
Essentially what we came up was fairly simple:
We'll want to take care that one evaluation does not block another evaluation.
I'd really like to run lorri watch
once and then not have to worry about it when I switch between projects. Is that safe, as designed? I think it'd work looking at where files end up, but I'm not 100% sure if that's incidental or on purpose.
I guess if it's on purpose, I'd also like a way to query status in my shell or editor—e.g. lorri status
figures out if the relevant process is idle, building, or in error. That way I can integrate it into my prompt. (edit: although I'd like that regardless of if it's a per-project or system daemon, so LMK if it would be more helpful for me to open another issue with more details to track separately?)
After @dudebout solved my problem in #6 of it watching too many paths, I'm encountering the opposite problem: it doesn't watch all paths it should.
A simple example:
let
pkgs = import <nixpkgs> {};
deps = pkgs.stdenv.mkDerivation {
name = "deps";
src = builtins.filterSource (path: type: baseNameOf path == "pkg") ./.;
installPhase = ''
echo "pkgs: with pkgs; [ $(cat pkg) ]" > $out
'';
};
in pkgs.stdenv.mkDerivation {
name = "test";
nativeBuildInputs = import deps pkgs;
}
This uses import-from-derivation (IFD) by first building a derivation for knowing the dependencies, depending on a file ./pkg
through builtins.filterSource
(touch
that file), then puts those dependencies in the build inputs. Lorri ends up not watching the file, as seen from the log:
[2019-04-01T01:01:22Z DEBUG lorri] Input options: Arguments { verbosity: 0, command: Watch }
Started
[2019-04-01T01:01:22Z DEBUG lorri::build_loop] original paths: 518
[2019-04-01T01:01:22Z DEBUG lorri::build_loop] -> reduced to: 3
[2019-04-01T01:01:22Z DEBUG lorri::build_loop] named drvs: {
"shell": "/nix/store/gc1303z31vq35sl044fxfk0z9pmdkg42-test.drv",
"shell_gc_root": "/nix/store/vl1rb2fsk4f50p5d6h2qsqj8iin86b20-lorri-keep-env-hack-test.drv"
}
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/gc1303z31vq35sl044fxfk0z9pmdkg42-test.drv" to "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-attr-shell"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/vl1rb2fsk4f50p5d6h2qsqj8iin86b20-lorri-keep-env-hack-test.drv" to "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-attr-shell_gc_root"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/s11p9yzvbiff29m1qr30kk7hpcgij5cl-lorri-keep-env-hack-test" to "/home/infinisil/.cache/lorri/lorri/gc_root/build-0"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/build-0" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-build-0"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/Test/lorri/shell.nix"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/Test/lorri"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/.config/nixpkgs/config.nix"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/.config/nixpkgs"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/.config/nixpkgs/overlays"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/.config/nixpkgs"
Completed(
BuildResults {
drvs: {
0: "/home/infinisil/.cache/lorri/lorri/gc_root/build-0"
},
named_drvs: {
"shell": "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell",
"shell_gc_root": "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root"
}
}
)
Meaning I can change the contents of the file ./pkg
, which would have an influence on nix-shell
, but lorri doesn't budge. And because there's no way to either forcefully include a file or forcefully reload everything, the only way to have it change the environment is by restarting lorri watch
(or changing a different file).
While this may sound rather theoretical, that's actually how almost all functions in nixpkgs for developing Haskell packages work, including developPackage
, callHackage
, callCabal2nix
, and more.
I am encountering this with my project (after an adjustment). Namely it doesn't reload when I add a dependency to the .cabal
file, which is something you do a lot in Haskell.
Hello. I suspect there must be some issue with README instructions but I can't really say what exactly. I'm using NixOS
and was following nixos instructions in README.
Issue is with eval "$(lorri direnv)"
which fails with an error.
eval "$(lorri direnv)"
watch_file: command not found
which then breaks completely:
$ lorri direnv
lorri: command not found
$ eval "$(lorri direnv)"
lorri: command not found
In a new terminal
$ lorri direnv
EVALUATION_ROOT="/home/marek/.cache/lorri/planning-game/gc_root/build-0"
watch_file "$EVALUATION_ROOT"
function declare() {
if [ "$1" == "-x" ]; then shift; fi
function punt () {
:
}
function prepend() {
varname=$1 # example: varname might contain the string "PATH"
# drop off the varname, so the remaining arguments are the
# arguments to export
shift
# set $original to the contents of the the variable $varname
# refers to
eval original=\$$varname
# effectfully accept the new variable's contents
export "$@";
# re-set $varname's variable to the contents of varname's
# reference, plus the current (updated on the export) contents.
eval $varname=\$$varname:$original
}
# Some variables require special handling.
#
# - punt: don't set the variable at all
# - prepend: take the new value, and put it before the current value.
case "$1" in
# vars from: https://github.com/NixOS/nix/blob/92d08c02c84be34ec0df56ed718526c382845d1a/src/nix-build/nix-build.cc#L100
"HOME="*) punt;;
"USER="*) punt;;
"LOGNAME="*) punt;;
"DISPLAY="*) punt;;
"PATH="*) prepend "PATH" "$@";;
"TERM="*) punt;;
"IN_NIX_SHELL="*) punt;;
"TZ="*) punt;;
"PAGER="*) punt;;
"NIX_BUILD_SHELL="*) punt;;
"SHLVL="*) punt;;
# vars from: https://github.com/NixOS/nix/blob/92d08c02c84be34ec0df56ed718526c382845d1a/src/nix-build/nix-build.cc#L385
"TEMPDIR="*) punt;;
"TMPDIR="*) punt;;
"TEMP="*) punt;;
"TMP="*) punt;;
# vars from: https://github.com/NixOS/nix/blob/92d08c02c84be34ec0df56ed718526c382845d1a/src/nix-build/nix-build.cc#L421
"NIX_ENFORCE_PURITY="*) punt;;
*)
export "$@"
;;
esac
}
export IN_NIX_SHELL=1
. "$EVALUATION_ROOT"
unset declare
but again evaluating this doesn't work and then breaks lorrri completely:
$ eval "$(lorri direnv)"
watch_file: command not found
$ eval "$(lorri direnv)"
lorri: command not found
We should make sure our locked distribution and binary releases don’t contain known security problems https://github.com/RustSec/cargo-audit
Blocked on #44
Had a chance to poke at lorri this morning; started following the README and tried to install the required direnv. It failed to build.
I'll admit that I don't fully grok nix quite yet, so it's entirely possible that this is my fault. Reporting to both get help, and in case the solution helps the next person in my scenario.
Relevant (I think) parts:
---
Installed Go for darwin/amd64 in /private/tmp/nix-build-go-1.11.5.drv-0/source
Installed commands in /private/tmp/nix-build-go-1.11.5.drv-0/source/bin
The binaries expect /private/tmp/nix-build-go-1.11.5.drv-0/source to be copied or moved to /nix/store/mkcsimqjfw9s2b9rn77yvx5bpfgavpp2-go-1.11.5/share/go
running tests
##### Testing packages.
go proxy: no archive z.1 v1.2.0
--- FAIL: TestScript (0.00s)
--- FAIL: TestScript/gcflags_patterns (1.60s)
script_test.go:151:
# -gcflags=-e applies to named packages, not dependencies (0.150s)
# -gcflags can specify package=flags, and can be repeated; last match wins (0.204s)
# -gcflags can have arbitrary spaces around the flags (0.045s)
# -ldflags for implicit test package applies to test binary (0.318s)
# -ldflags for explicit test package applies to test binary (0.351s)
# -ldflags applies to link of command (0.145s)
# -ldflags applies to link of command even with strange directory name (0.170s)
# -ldflags applies to current directory (0.183s)
# -ldflags applies to current directory even if GOPATH is funny (0.000s)
> [windows] cd $WORK/GoPath/src/my/cmd/prog
> [darwin] cd $WORK/GoPath/src/my/cmd/prog
FAIL: testdata/script/gcflags_patterns.txt:49: directory $WORK/GoPath/src/my/cmd/prog does not exist
FAIL
FAIL cmd/go 165.022s
ok cmd/vet/internal/cfg 0.024s
2019/03/31 14:15:52 Failed: exit status 1
builder for '/nix/store/ha1afyw2wnna7024dajzssx91ra0ivpa-go-1.11.5.drv' failed with exit code 1
cannot build derivation '/nix/store/ls2dcrkmdrx0jdadw013fjxs0gk87v6z-direnv-2.19.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ls2dcrkmdrx0jdadw013fjxs0gk87v6z-direnv-2.19.2.drv' failed
According to the description in the current rust book and the issues tracked by rust-lang/rust#53125, the preferred way of doing nested modules in newer rust versions is the directory/filename combination.
So instead of
foo/bar/mod.rs
foo/bar/abc.rs
where you have a module foo::bar
(implemented in foo/bar/mod.rs
) and a module foo::bar::abc
(implemented in foo/bar/abc.rs
), you would do
foo/bar.rs
foo/bar/abc.rs
where foo:bar
is implemented in in foo/bar.rs
.
Actual users need to be asked what their shell environments look like, and we need to integrate these things into tests and examples.
Requirements should be collected in this issue.
@grahamc writes:
Repro: https://github.com/tweag/lorri/tree/missing-source-file-panic-repro/repros/missing-source-file-panic
We should filter out paths which don't exist, and not attempt to watch them.
We should also pass through the error log messages.
@grahamc writes:
Projects that use IFD don't get a great experience, because in the tree:
IFD-build-tools -> IFD-build-result -> shell-environment
the shell environment and the IFD build result are rooted, but the build tools are not rooted.
In the keep-IFD-envs branch, some work is done to pin those too, but it has problems with paths >not being realized.
Currently each CI run builds all transitive rust dependencies. Cargo has no concept of output sharing, but Mozilla devised a ccache
like thing called sccache
to cache build artifacts.
There is experimental Rust support:
If you are using Rust 1.18 or later, you can ask cargo to wrap all compilation with sccache by setting RUSTC_WRAPPER=sccache in your build environment.
We should set it up to push to a Travis cache that is shared between builds.
Using shell.nix
with import <nixpkgs> {};
mkShell {
shellHook = ''
export FOO=BAR
'';
}
I don't find FOO
set when using lorri
.
@Profpatsch writes:
This is a major trick to prevent unnecessary evaluation:
Before actually starting evaluation, we should transform the nix file syntax to a normal form so that whitespace changes don’t lead to actually starting the (expensive) nix evaluator.Reformatting
[a b c ]
to
[ a b c ]
should not lead to an evaluation.
The importance of this should become clear with a simple example. On my laptop, a small
shell.nix
likewith import <nixpkgs> {}; mkShell { buildInputs = [ ghc ]; }
takes more than five seconds to evaluate, at 100% CPU.
My editor saves on focus lost.
@grahamc writes:
This would be a major undertaking, because our current evaluation tracking is based on the file system watcher. This means that we don't actually have any understand what is in the file, just when the file changed. however, I do understand the value here. I just think it would be difficult to do, and maybe a project for down the road.
$ lorri -v watch
Started
thread '<unnamed>' panicked at 'Builder failed: AddRoot(Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }))', src/
build_loop.rs:81:29
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any', src/libcore/result.rs:1009:5
With debug logging, I found out this was due to missing /nix/var/nix/gcroots/per-user/$USER
(https://github.com/target/lorri/blob/master/src/roots.rs#L32) which seems to happen on a single-user non-NixOS install of nix.
@Profpatsch writes:
When I have a shell.nix in a directory
/path/to/something/examples
, the evaluation result and gc root looks like this:EvaluationResults { drvs: { 0: "/home/philip/.cache/lorri/examples/gc_root/evaluation-0" }, named_drvs: { "shell_gc_root": "/home/philip/.cache/lorri/examples/gc_root/attr-shell_gc_root", "shell": "/home/philip/.cache/lorri/examples/gc_root/attr-shell" }
~/.cache/lorri/
is the XDG cache dir set up by lorri, but the project is only reflected by the last path segment of its path, in this caseexamples
.This assumes parent/project folders names never have the same name, which I think is not an assumption we can make (like in this example, multiple projects could have a folder named examples, leading to a broken cache if both are referenced at the same time).
Since we don’t need to share the cache, we could mangle the full path, something like
~/.cache/lorri/RpathStoSsomethingSexamples/
Unlike vim we should actually escape the special character and make the thing injective …
.replace("/", "%").replace("%", "%%")
@zimbatm writes:
in direnv I use
hexencode(sha256(full_path))
as the cache key, it works pretty well
@Profpatsch writes:
Hashes are nice, but modern tools have a tendency to overuse hash functions in my opinion. :P
@zimbatm writes:
The nice thing is that it removes whole classes of potential problems:
- weird file names, no need to worry about escaping (btw, your replace functions should probably be inverted)
- paths being too long for weak OS or filesystems
I have this .envrc
:
cp gemset.nix tmp-gemset
bundix --gemset=tmp-gemset
if diff tmp-gemset gemset.nix; then
echo No change to gemset.nix
rm tmp-gemset
else
echo Updating gemset.nix
mv tmp-gemset gemset.nix
fi
use lorri
watch_file Gemfile.lock
The idea is that, if Gemfile.lock
changes, I want to run bundix to generate a new gemset.nix
, which my shell.nix
depends on.
What I'm finding is that lorri runs nearly every prompt. I think that what happens is that the gemset.nix
gets read in the process which generates an inotify event? But I'm not 100% on this particular and not sure how to investigate.
We should make it clear to the user why every evaluation we use happens. This also helps debugging.
I envision this as our watching module exporting a Reason
type with detailed information about why the change was triggered, which we display before the evaluation.
I just reinstalled nix,
and installed lorri
, cleaned the ~/.cache/lorri
but now I am getting:
transforms-0.3.5.drv\'",
"instantiated \'pandoc-2.7.1.tar.gz\' -> \'/nix/store/wwzxk03q9vhj60x0v6wma3s6kgb3n6h3-pandoc-2.7.1.tar.gz.drv\'",
"instantiated \'pandoc-2.7.1\' -> \'/nix/store/j9x3j0ljd3anbj03fx0nzvxn8dhpbgyp-pandoc-2.7.1.drv\'",
"instantiated \'grimoire\' -> \'/nix/store/rwksvqhhkl0fpjlfish8b2jx6cjq76sh-grimoire.drv\'",
"error: path \'/usr/bin\' is not in the Nix store, at (string):47:12",
"(use \'--show-trace\' to show detailed location information)"
]
}
)
nix-shell
works the way it should though.
So this is really a Nix issue (how to triage gc roots across a system?), but now that I am spawning lorri's everywhere to watch my things I'm finding that managing my gcroots is a bit of a pain. Firstly looking up the paths corresponding to a given project (or the other way around) is not rocket science but introduces enough of a disconnect to cause friction. Introducing 3 gcroots per project is also a factor,
making a list of gc roots seem more intimidating than it really is :).
I don't mean to complain, but this seems like something that could have a good solution (perhaps) as part of lorri and for everyone's benefit.
Any plans/thoughts on this front?
I suppose a basic "here is the gc roots for this project" as entries in lorri info
would be a start,
and perhaps a simple "clear the eval cache" (phrased better) subcommand?
@grahamc writes:
Not 100% sure how yet, maybe with expect, but verify that lorri + direnv + the background evaluator works properly.
Note: This is just for the first few test cases. The results of this ticket should also produce a system we can use for further testing, so we can expect bug fixes to come with test cases.
Then we can make editor integration nicer.
Strange that I have to install lorri
with nix-env
instead of just tossing it in my configuration file as environment.systemPackages = [ lorri ];
. Is there going to be an upstream package in the future?
I tried making my own like this:
{ config, pkgs, ... }:
let
lorri = pkgs.stdenv.mkDerivation rec {
pname = "lorri";
version = "rolling-release";
src = pkgs.fetchFromGitHub {
owner = "target";
repo = "lorri";
rev = "094a903d19eb652a79ad6e7db6ad1ee9ad78d26c";
sha256 = "11i4kbwclwyvznyd9q69fq36fjasvs72ziz0555hl3fjbbq0n72q";
};
};
in
{
environment.systemPackages = [ lorri ];
}
But it failed like this:
shkhgcxs3aljqz771rilj8c-nixpkgs-18.09.2436.395a543f360/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating the attribute 'system.activationScripts.script' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/activation/activation-script.nix:67:9:
while evaluating 'textClosureMap' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/strings-with-deps.nix:70:35, called from /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/activation/activation-script.nix:88:18:
while evaluating 'id' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/trivial.nix:14:5, called from undefined position:
while evaluating the attribute 'text' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/activation/activation-script.nix:9:5:
while evaluating the attribute 'text' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/strings-with-deps.nix:77:38:
while evaluating the attribute 'sources' of the derivation 'etc' at /nix/store/yn6df0qzkshkhgcxs3aljqz771rilj8c-nixpkgs-18.09.2436.395a543f360/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/system/etc/etc.nix:20:20, called from undefined position:
while evaluating the attribute 'source' at undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:75:45, called from undefined position:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:336:9:
while evaluating the option `environment.etc.crontab.source':
while evaluating the attribute 'mergedValue' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:369:5:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:369:32, called from /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:369:19:
while evaluating 'check' at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/types.nix:240:15, called from /nix/var/nix/profiles/per-user/root/channels/nixpkgs/lib/modules.nix:370:10:
while evaluating the attribute 'allFiles' of the derivation 'crontabs' at /nix/store/yn6df0qzkshkhgcxs3aljqz771rilj8c-nixpkgs-18.09.2436.395a543f360/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating the attribute 'text' of the derivation 'system-crontab' at /nix/store/yn6df0qzkshkhgcxs3aljqz771rilj8c-nixpkgs-18.09.2436.395a543f360/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating the attribute 'passAsFile' of the derivation 'system-path' at /nix/store/yn6df0qzkshkhgcxs3aljqz771rilj8c-nixpkgs-18.09.2436.395a543f360/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
illegal name: '.drv'
Multiple people have had problems with setting up lorri that boiled down to them trying to source lorri direnv
outside of the .envrc
file and missing the direnv stdlib (sometimes even in non-bash shells). One example is #59.
We must print a clear warning in that case.
Open question: how to best recognize whether lorri direnv
is called from direnv
? cc @zimbatm
@grahamc writes:
In Direnv, we'll want to be able to get an event when the shell is unloaded, for #32.
An important question is: can editors handle exit hooks too?
This is pretty important behavior, I think. A future where lorri helps with service management could also benefit from these exit hooks.
@zimbatm writes:
Because the direnv process is only called on prompt change it makes this quite difficult to implement in a reliable manner. The user can exit or kill the shell, or exec into a different program. All of these would not trigger the unload hook.
See direnv/direnv#129 for some discussions around the topic.
@shlevy writes:
Specifically for #32, wouldn't a CLOEXEC fd handle both shell exit and execing into a different program?
@zimbatm writes:
this would imply that the fd is attached to the shell. direnv itself is only executed shortly before a prompt.
@shlevy writes:
https://github.com/taviso/ctypes.sh
More seriously I'm wondering if we can just take the hit of "you have to run your shell under lorri if you want certain features" so we can take care of any setup like this
@zimbatm writes:
direnv is not that complicated to re-implement. One route would be to have
lorri shell
only support bash, and re-implement a direnv-light under it. Sincelorri shell
would be staying around it can control the daemon liveness (and have one running per shell).The biggest downside is that each IDE would have to be extended to support the environment live-reloading mechanism. Maybe it's Ok to only support VSCode out of the box?
@shlevy writes:
IIRC the main point of using direnv was getting multiple editors/shells supported for free.
@grahamc writes:
Yes, direnv is like a secret weapon of "how do I make this work with every editor." Thinking further, this is not a dealbreaker. We can use
lorri shell
for service management or something. Having a background evaluator going will do, essentially, nothing: it is just sleeping and waiting for an event from the kernel.
@zimbatm writes:
Another idea:
add a
use_lorri
function to the direnv stdlib. This will be a good excuse for me to introduce a mechanism to extend the stdlib.use_lorri() { # select a place where to put the eval output local lorri_path lorri_path=$(direnv_layout_dir)/lorri # lorri would output all the files that have been used and are not in the /nix/store # lorri would write the eval output to "$lorri_path" files=$(lorri evaluate --list-mutable-files --output "$lorri_path"`) # load the lorri evaluation output. here it's in the .env format. dotenv "$lorri_path" # instruct direnv to reload if any of these files have changed for file in $files ; do watch_file "$file" done }
@aspiwack writes:
Pinning nixpkgs is very underused. Mostly because it's a rather painful process (one has to find the commit reference for the current nixpkgs then remember the code snippet to copy/paste in).
So I imagine that lorri could have a command to do all these things for me.
Here is a straw-man proposal:
Pin the project to the current nixpkgs version:
$ lorri pin
Pin the project to the latest nixpkgs-stable:
$ lorri pin nixpkgs-stable
Update the pinned version to the current nixpkgs:
$ lorri pin --update
I'm not sure what it should look like nix-expression-wise. Certainly the pinning action should be in file checked into the version control system. Possibly by modifying the shell.nix file. Or possibly by making a lock.nix file under lorri's control, which you would require the user to call from shell.nix. Something to that effect.
@zimbatm writes:
for some prior art see:
@shlevy writes:
If we do this it would be good to coordinate with flakes if relevant.
For lazy users such as myself, could there be a subcommand which writes the stock .envrc
, does a build of shell.nix
, and provides further instructions? For example:
~/myproject $ lorri register
An .envrc for myproject does not exist, writing it now.
Building shell.nix...
these paths will be fetched (0.04 MiB download, 0.20 MiB unpacked):
/nix/store/4vv9dxgcjad248mwwfd2j80kss2cxxw8-hello-2.10
copying path '/nix/store/4vv9dxgcjad248mwwfd2j80kss2cxxw8-hello-2.10' from 'https://cache.nixos.org'...
The shell has been built and saved in ~/.cache/lorri/myproject.
Run `lorri watch' in another terminal to keep it updated.
If you have enabled direnv, then you should be ready to go!
direnv: loading .envrc
direnv: export +IN_NIX_SHELL ....
~/myproject $
So, lorri is amazing, but immediately after trying it, i found it lacking in one regard: I jump between projects a lot, and would have to remember to run lorri watch
.
So instead i wrote this little .direnvrc
:
use_nix() {
eval "$(lorri direnv)"
found=false
for lpid in $(pgrep -xf "lorri watch"); do
lpwd="$(readlink -e "/proc/$lpid/cwd")"
self="$(readlink -e "/proc/$$/cwd")"
if [[ "$lpwd" == "$self" ]]; then
found=true
fi
done
if [[ "$found" == "false" ]]; then
lorri watch 2>/dev/null 1>/dev/null &
pid=$!
echo "started lorri watch with pid $pid"
disown $pid
fi
}
I'm sure this could be written in a better way, but that's the quickest pure-shell version i could come up with right now.
I'd just like to have this functionality in lorri :)
When setting up direnv with lorri, it is a bummer to have it fail the first time due to no watch
runner. It would be cool if lorri direnv
would auto-evaluate the expressions just the first time.
One concern I have is this might encourage people to never watch
their project, and thus have ancient dependencies stick around for an eternity.
Running on Ubuntu (via nix-review for the nixpkgs PR) and ran into this. There is plenty of space on the device.
lorri watch
Started
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Unrecoverable(Notify(Io(Os { code: 28, kind: Other, message: "No space left on device" })))', src/libcore/result.rs:997:5
stack backtrace:
0: 0x5651e5047b6f - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h0783754d3fee6459
1: 0x5651e5057977 - std::sys_common::backtrace::print::h3122233e347d34d5
2: 0x5651e504b153 - std::panicking::default_hook::{{closure}}::h5dd267379284318d
3: 0x5651e504aed3 - std::panicking::default_hook::h91865806e9bd29bc
4: 0x5651e504b84f - std::panicking::rust_panic_with_hook::hb03b843366405f92
5: 0x5651e504b3e1 - std::panicking::continue_panic_fmt::h55006de900d00bf9
6: 0x5651e504b2c5 - rust_begin_unwind
7: 0x5651e507805c - core::panicking::panic_fmt::h86b59874e6b77e96
8: 0x5651e4efd5cd - core::result::unwrap_failed::ha54ccda9a5abc9c9
9: 0x5651e4f057ba - lorri::build_loop::BuildLoop::forever::he31eedd0e9c48593
10: 0x5651e4eee859 - std::sys_common::backtrace::__rust_begin_short_backtrace::h9ca173341b3c72e9
11: 0x5651e4eef458 - std::panicking::try::do_call::ha3006f6b706ee83b
12: 0x5651e5061c09 - __rust_maybe_catch_panic
13: 0x5651e4f1d267 - <F as alloc::boxed::FnBox<A>>::call_box::h019709e300d681a2
14: 0x5651e504693d - std::sys_common::thread::start_thread::hf9c1221451dff292
15: 0x5651e50511b5 - std::sys::unix::thread::Thread::new::thread_start::h34e50d5a3156c212
16: 0x7fc9261c8ef6 - start_thread
17: 0x7fc925ee822e - __GI___clone
18: 0x0 - <unknown>
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any', src/libcore/result.rs:997:5
stack backtrace:
0: 0x5651e5047b6f - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h0783754d3fee6459
1: 0x5651e5057977 - std::sys_common::backtrace::print::h3122233e347d34d5
2: 0x5651e504b153 - std::panicking::default_hook::{{closure}}::h5dd267379284318d
3: 0x5651e504aed3 - std::panicking::default_hook::h91865806e9bd29bc
4: 0x5651e504b84f - std::panicking::rust_panic_with_hook::hb03b843366405f92
5: 0x5651e504b3e1 - std::panicking::continue_panic_fmt::h55006de900d00bf9
6: 0x5651e504b2c5 - rust_begin_unwind
7: 0x5651e507805c - core::panicking::panic_fmt::h86b59874e6b77e96
8: 0x5651e4efd534 - core::result::unwrap_failed::h92869f304e469b67
9: 0x5651e4f0724a - lorri::ops::watch::main::hd6aa0621e18e8188
10: 0x5651e4ede961 - lorri::main::hb735b843cfb29abb
11: 0x5651e4edf282 - std::rt::lang_start::{{closure}}::h6844233bb9b63cd0
12: 0x5651e504b262 - std::panicking::try::do_call::h11736d510a2ae4e6
13: 0x5651e5061c09 - __rust_maybe_catch_panic
14: 0x5651e505ca9d - std::rt::lang_start_internal::h483ed2ec71424c9f
15: 0x5651e4edf274 - main
16: 0x7fc925e17b8d - __libc_start_main
17: 0x5651e4ede2c9 - _start
18: 0x0 - <unknown>
I've intalled NixOS module as per doc to enable direnv.
Then cloned lorri
, entered dir, but instead of installing lorri, i've run direnv allow
. It started to download stuff, but when it started compiling Rust, I decided to ^C it.
But I couldn't! After interrupt it restarted install of Rust, until I killed terminal.
I haven't looked at the implementation, but it would be helpful if following worked:
$ lorri watch -f nix/myfoo.nix --run ls
Motivation: hercules-ci/arion#36
@Profpatsch writes:
We have two modules which are named nearly the same and both shell out to nix-build, but do different things.
We should either change the names, or move them into one module (or into two modules in the same file).
At the moment we just print the events to the lorri watch
stdout, the user has no idea how much needs to be built and what nix does under the hood.
Since lorri should be a small wrapper around nix, we should output exactly the same information as nix.
@Profpatsch writes:
Plugins like
cargo audit
provide helpful functionality to the project.We cannot add them to
Cargo.toml
’s[dev-dependencies]
, because that requires their dependencie’s versions to align with the versions of our other dependencies. They need to be installed withcargo install
and then provided to the project somehow.Ideally, we could integrate them into our
shell.nix
environment, by ways ofcarnix
for example.
@shlevy writes:
Does upstream consider this a deficiency in the cargo spec?
@Profpatsch writes:
I don’t know.
They might just assume users will set it up globally with cargo install somehow.
@Profpatsch writes:
There is a new rust->nix tool on the block: https://github.com/kolloch/crate2nix
It might be nicer to use than carnix.
How do I use lorri
to obtain a pure
environment in the nix-shell
sense?
@grahamc writes:
The background evaluator will need to listen for new projects to watch. To do so, the evaluator should listen on a unix socket at a Well Known Location (see XDG spec for where -- not sure which subsection at this time) new projects. The API should also support "cancel" messages, to stop watching a project.
Note, because of the nature of direnv -- many shells opening the same project -- we'll want to implement a refcount like system, and only "cancel" the evaluation when the last shell closes.
Blocked on #33
@zimbatm writes:
Another approach would be to "keep-alive" the daemon:
the user enters the project folder:
lorri evaluate --folder $PWD
is called
if$XDG_RUNTIME_DIR/lorri/daemon.sock
doesn't exist (or isn't alive), start the daemon
send a message watch/path/to/project
to the daemon
The daemon is started with a default TTL per project. If the TTL is reached, stop watching the project. If no projects are to be watched, shutdown the daemon and remove the socket.Whenever files are changed in the project, extend the TTL.
The only issue is if the user sits too long in the project, with no files changing. Let's say the user goes to lunch and comes back to his seat. In that case the user would still be in the shell but the daemon has shutdown. This could be solved by running
lorri evaluate --folder $PWD
on each prompt, but then it means that lorri needs to hook into the shell as well.
@shlevy writes:
Just mentioned this on #33, but why don't we just treat socket close as the "cancel" command?
We should disable all optional features of dependencies we don’t need, reducing build times and CI overhead.
Tools like
https://github.com/kbknapp/cargo-graph and
https://github.com/sfackler/cargo-tree
are helpful to visualize what the tree looks like and what can be pruned.
Blocked on #44
I forgot to post my findings, but yes this is true! Nix doesn't emit /path/to/default.nix
as the true file, it emits just '/path/to. A possible solution here is to append a
/default.nix` when an imported directory is identified.
Originally posted by @grahamc in #21
We currently watch the whole directory if a user does import /path/to/directory
. Nix implicitely assumes /path/to/directory/default.nix
if used with import
. We should mirror that, to prevent watching too many files.
The mdsh project gives us a way to run code blocks embedded in markdown files.
We should use this to test the README and our tutorial automatically in CI, to make sure they don’t go out of date.
I already patched mdsh to work with CI in zimbatm/mdsh#16.
direnv: loading .envrc
./.envrc: line 66: /home/michael/.cache/lorri/example/gc_root/build-0: No such file or directory
direnv: export +IN_NIX_SHELL
Seems like this should be detectable!
Set up a project to work with lorri and direnv.
For now that means:
shell.nix
exists and is a valid shell file.envrc
If the files don’t exist, we simply create them and populate them with our setup code. If they exist, we don’t touch them and print an informational message, like:
shell.nix
exists, skipping. Make sure it is of a form that works with nix-shell
.”.envrc
exists, skipping. Please add eval "$(lorri direnv)"
to it to set up lorri support.”In the case of shell.nix
we could also try to find out whether it is a correct shell file.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.