Comments (4)
This is hard to support because the NAR file format puts the size of the file before its contents. So if we can't rely on the file size reported by lstat()
, we would have to read the entire file into memory first, which would be a problem for supporting large files.
Related: #10019
from nix.
I see. Here's a few thoughts I have on how this could theoretically be solved, note that I don't really know how feasible these are given that I'm not familiar with how Nix works internally:
- Read the file as we do now, but then attempt to continue reading. If this returns more data than expected, keep adding to the NAR file and then seek back to the beginning of the file, updating the size.
- This wouldn't work if the NAR file is created in a streaming fashion.
- Buffer the entire file into memory or a temporary file if and only if the filesystem reports a size of zero and a read call yields data.
- This would still break for filesystems that report a non-zero size that is smaller than the actual file size.
Some things to consider:
- What if the filesystem reports a size larger than the actual file? Would this lead to an invalid NAR file being generated, or an error being emitted? We may be able to employ a strategy similar to what I've first suggested above.
- How to handle infinitely large files? What should we do if someone attempts to use, for example,
/dev/random
? Should we have a hard limit on how large a file can be? Should we have this limit only for files that report a zero-size?
from nix.
Triaged in Nix team meeting:
-
@roberth: we could special-case the 0. if the size is reported as 0, we could still try reading it and buffer it.
- @edolstra: at least that would work. wonder what POSIX has to say what should happen if the file size is not reported correctly (these files do exist, such as
/proc
)
- @edolstra: at least that would work. wonder what POSIX has to say what should happen if the file size is not reported correctly (these files do exist, such as
-
@edolstra: we used to buffer essentially entire files in memory, but that simply doesn't scale
- it could be done, but it's hard to do in constant-space memory
-
This wouldn't work if the NAR file is created in a streaming fashion.
@Ericson2314: It is created in a streaming fashion, we wish to use constant space and often create NARs directly into a sink like a pipe or socket.
-
This mismatch between actual file size and reported file size is considered absolutely legal under Linux (citation needed)
@Ericson2314: We would love to see that citation :)
from nix.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-05-15-nix-team-meeting-minutes-146/45491/1
from nix.
Related Issues (20)
- `builtins.fetchTarball` with a single-file tarball produces an empty derivation HOT 3
- fetcher cache sqlite error: table Cache has no column named domain HOT 1
- "Alternate FOD" for fixed output derivation that fails, mirror-like behavior
- Settings aliases are not supported as flags, while their "real" names are HOT 1
- nix-prefetch-url: support SRI hashes directly HOT 2
- Error: could not set permissions on '...' to 755: Operation not permitted when using experimental local overlay store HOT 4
- "attributes elided" placeholder's count can underflow
- Machine readable nix why-depends
- `SourceAccessor::readFile` should be streaming in constant memory
- [2.23.1] nix sandbox broken on darwin HOT 5
- Upgrading to current nix master corrupted my db HOT 1
- DNS resolution breaks on NixOS when having static resolv.conf HOT 3
- Outputs from another flake is current flake's outputs HOT 5
- Nix remote store commands (ping, copy etc..) don't work with Fish shell HOT 6
- Shallow fetches of specific revs do not work with all servers
- Installing darwin specific installer fails HOT 1
- FFI issue: missing `flake/flake.hh` in API exposed in `include/nix` HOT 3
- C API: document error handling HOT 1
- Nix documentation does not describe how to set up a multi-user nix installation on a new user HOT 1
- Passing store paths into functions from other flakes causes errors in pure evaluation mode
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nix.