Comments (6)
@KiaraGrouwstra what you write indicates to me that the mechanism you desire may indeed be more suitable to be implemented on top of Nix rather than as part of it, for example as part of a (still hypothetical) Nixpkgs CLI.
I have a hard time imagining how caching derivations would help, because all that gives you are more opaque store paths. What we'd really need for actionable suggestions is information about Nixpkgs recipe parameters, and we simply don't have them at the Nix layer. The derivation's name
attribute is a clumsy proxy for actual metadata.
Another thing that could get us closer to your vision may be a generic cache mechanism where one adds an attribute set to the argument to derivation
, which is serialised (only) into the cache. Then consumers could figure out what to do with that. But the implications of this would need to be considered properly.
from nix.
Triaged in Nix team meeting:
-
@edolstra: there is no way we can implement that, because the evaluation model does not allow that
maybe package foo with this version x of dependency bar is not in the cache, but using version y it might have been.
Nix cannot answer that question. It doesn't know what's in the cache, and cannot map that back to arguments or attributes Nixpkgs
-
@fricklerhandwerk if anything, this could only (in principle) addressed at the Nixpkgs layer, although even then it's not clear how exactly that would work without combinatorial explosion
-
decision: closing it since we don't know how to implement it. feel free to re-open with a concrete proposal.
from nix.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-05-15-nix-team-meeting-minutes-146/45491/1
from nix.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/overlaying-packages-using-cuda-to-use-zluda/45374/5
from nix.
i thought about this a bit more. @fricklerhandwerk
i think caches currently not having the info needed for this means we could address this for the future by allowing caches to start storing not only a hash's build results, but also the (instantiated) derivation that produced that result, as we do in /nix/store
.
while this could yield a trade-off with privacy (-> make this opt-in), i think that should take care of the info needed to answer such queries for future cache pushes. (having this info in the cache could further bring verifiability to build results.)
i think the concern around combinatorial explosion could be alleviated by:
- making this opt-in to the cache
- making this opt-in to the pushing user
- potentially limiting this further from derivations to packages specifically (i.e. has
pname
) which i think is the 20/80 - the ability to then group/filter cache hits by e.g. such a
pname
if not optionally also version string/rev - while a limitation, potentially again a 20/80 - requiring a manual action on the part of the user to request a miss explanation for a given derivation (/package)
- the number of cached versions of a package in practice already being limited by the cache actually having to store all their cached results
- potentially offloading work regarding the explanation to a client running at the requesting user, limiting the cache's work on such a request to sending instantiated derivations with cached build results for a given
pname
(+ version) - summarizing likely reasons by presenting the user with a ranking of tallied key paths that differ between the requested vs. cached derivations. i.e., 3 cached derivations used a different value for
foo.bar
, 2 additionally had a value for keybaz
that is missing in the requested derivation. if the user picks such an acceptable difference, they could then be presented with top similar cached derivations (with similarity measured in e.g. number of key paths with differing values).
i hope that i properly understood the raised concern, and that this might go some way toward addressing it.
from nix.
@fricklerhandwerk thanks for your response.
on the derivations, i think you're right my terminology was off - i guess i intended to say the cache should know the evaluated (json?) input that would bring about the cached build result.
i agree a generic caching mechanism could help facilitate this, given a nix cli (i think this isn't specific to nixpkgs) and cache implementing it.
from nix.
Related Issues (20)
- copying '«unknown»/' to the store HOT 2
- Why does `nix-daemon.service` set `KillMode=process` HOT 7
- Missing documentation: `unsafeDiscardOutputDependency`
- Nix integer overflow is implemented by undefined behavior HOT 1
- Enable C++ Sanitizers for Nix HOT 3
- Reorganize the language reference HOT 3
- C API: `nix_alloc_primop`'s `user_data` leaks
- nix flake show should show package descriptions if available HOT 1
- Hacking guide link in the readme doesn't exist HOT 1
- config.h is missing following the hacking guide: HOT 2
- Failed installation on Ubuntu 22.04 1 LTS HOT 1
- Test build-remote-trustless-should-pass-2.sh fails. (Filename Too Long) HOT 1
- `builtins.fetchTarball` with a single-file tarball produces an empty derivation HOT 3
- fetcher cache sqlite error: table Cache has no column named domain HOT 2
- "Alternate FOD" for fixed output derivation that fails, mirror-like behavior
- Settings aliases are not supported as flags, while their "real" names are HOT 1
- nix-prefetch-url: support SRI hashes directly HOT 2
- Error: could not set permissions on '...' to 755: Operation not permitted when using experimental local overlay store HOT 4
- "attributes elided" placeholder's count can underflow
- Machine readable nix why-depends HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nix.