Comments (8)
@thoughtpolice are you installing Buck2 by writing a Nix derivation? If we included that Nix derivation in the root of the repo, would that make it easy in Nix world?
Yes, this Nix derivation is here. There are one or two ugly hacks inside of it, but otherwise it is able to be built like (almost!) any other Rust program in Nixpkgs (which is the big library of Nix code that contains the set of things that Nix, the package manager, can install for you): https://github.com/thoughtpolice/buck2-nix/blob/main/buck/nix/buck2/default.nix
You might notice I also rename the binary from
buck2
tobuck
. Slightly heretical I admit, but I'm not a user of the original Buck so I can get away with it :)
It would make it somewhat easier, but I hesitate to just add it, because:
- A normal Nix expression isn't enough in this case; we would likely want Buck to have its own "Nix Flake". Flakes are sort of an abstraction on top of Nix, but they allow some fun features.
- It's not much more code in any case, so I wouldn't be scared.
- It would let you do some cool things, for example any Nix user anywhere could simply run
nix run facebook/buck2 -- --help
(assuming buck2 eventually graduates to @facebook proper) and get a working copy ofbuck2
to play with! Yes, it's (really!) that easy.
- It would need to be CI'd in order to stop regressions
- But that means it would be very annoying for non-Nix users to fix failures
- Nix is not really an easy tool to debug if you haven't used it
- e.g. the protoc-vendored patch would have been completely held up on it
- But most of all, we will need a working
Cargo.lock
file in order for Nix to build buck2.- This is effectively a must-have requirement.
- Right now, I have a shell script that updates the
buck2
Nix expression, and every time it does this, it re-generatesCargo.lock
and includes it.- Enjoy and please excuse the 1-letter Haskell variable names: https://github.com/thoughtpolice/buck2-nix/blob/0ec7878847a7c4cc811ef302f5ca31ac45c47938/buck/nix/update.sh#L80-L124
- My understanding is that Meta does not use lockfiles for its Rust projects, precisely because there is already a globally consistent view of all Rust dependencies inside your main repository (thanks to vendoring things and letting Buck itself drive the build). Therefore there is no need, because when you write
needed-foo-dep = "0.8"
, there is no ambiguity about what it refers to (e.g. 0.8.1, 0.8.2, etc) — it refers exactly to the version ofneeded-foo-dep
inside the repository itself.
- Inevitably, one day, a
buck2
package will also appear upstream in Nixpkgs. This is perfectly OK and can live alongside any Nix expressions located in this repository (the Nix code here is used for development, the Nix code in Nixpkgs used to distribute things to users.) Not this day, but one day...
At the moment, I'd be happy to contribute one, but at the pace buck2 is moving before its stabilization I'm not sure if y'all want all these burdens. And the lockfile one is absolutely mandatory, but also seems to affect other projects like Sapling, where Nix developers have had to generate our own lockfiles and include them. So I don't know what the policy should be here.
That said — I find Nix to be wonderful to adopt in some sense, for projects like this, because I find it helps encourage codebases to follow best open source practices; when in Rome, do as the Romans. So even if it slows you down sometimes, it may help "keep you honest" e.g. you can't pull hacks like "oh I'm going to download something over the internet during the build", you have to explicitly declare all your needed deps, stuff like that. Stuff I think you probably certainly agree with!
I'd be happy to think about writing and helping maintain an upstream Nix derivation, but maybe it would be easier to hash out over a video call or something. Unfortunately I'm incredibly busy at least until Saturday, at this rate, so it would probably have to wait for Monday or Tuesday. Maybe something to think about!
from buck2.
I think having all Nix users forced to carry a patch would be very sad, so definitely don't want to do that.
In a previous incantation, if PROTOC or PROTOC_INCLUDE were already set, we left them alone and didn't change them. That would be one thing to do, Thoughts @stepancheg ?
I also wonder if on Nix the protoc_bin_vendored should return an Err? Or maybe that's too hard to detect reliably.
from buck2.
Detecting when Nix is being used is probably not easy or reliable. Well, you could just ask if /usr
exists, or you could try to just execute protoc --version
and see if it fails, which is a pretty good, but...
It might also be worth just making the usage of protoc-bin-vendored
a feature in the Cargo file, that can be disabled or enabled, I think? That way you have to opt into it pretty explicitly without just setting some magic environment variables that are ambient and not easily discoverable.
Please note that I have not tried to self-bootstrap buck2 with buck2 (I don't know if that will be the "recommended" way to build production binaries in the long run, once remote execution and caching are in place, for example) so I don't know if this might be impacted, but from reading previous commit messages, I think y'all have that already handled regardless.
from buck2.
PROTOC or PROTOC_INCLUDE were already set, we left them alone and didn't change them. That would be one thing to do
Should be OK, if we rename them to BUCK2_PROTOC
and BUCK2_PROTOC_INCLUDE
to avoid accidentally reading environment variables we are not supposed to.
on Nix the protoc_bin_vendored should return an Err? Or maybe that's too hard to detect reliably.
It is not the job of the crate to do it. It just returns the binary packed in the crate, that's it.
patchelf
Patchelf seems to be too hard to do reliably (because if has to do it inside the dependency crate).
Some people in the internet suggest using nix-ld. Can't it be used? I cannot understand it from the page description.
from buck2.
My view is that working out the box is a good thing, so that means having protoc-bin-vendored be default is good, as it will reduce a lot of friction for Windows/Ubuntu users. Having that overridable by environment variables also sounds good (not too fussed on their names). Detecting Nix, running patchelf or a custom nix-ld all seem to go against the Nix philosophy, so seem less desirable.
@thoughtpolice are you installing Buck2 by writing a Nix derivation? If we included that Nix derivation in the root of the repo, would that make it easy in Nix world?
from buck2.
Some people in the internet suggest using nix-ld. Can't it be used? I cannot understand it from the page description.
nix-ld
is clever but inappropriate. The assumption behind it is that you are running NixOS, and thus a path like /usr/lib
does not exist. It then mounts a fake FUSE filesystem onto these paths, and intercepts calls for .so
libraries. So, for one, it requires a FUSE daemon to exist, so that the filesystem can translate paths like /lib64/ld-linux-x86-64.so.2
into a different path in the /nix/store
. And, this assumes you're running NixOS, but to be clear I am not; I am using normal Ubuntu Linux, with Nix (the package manager) installed on top of it. I do actually have a real /lib64/ld-linux-x86-64.so.2
on my system. So why is this all a problem?
This is because when you tell Nix to build something, it creates a hermetically sealed filesystem namespace under a different (random) Unix user, then runs the build process (e.g. cargo build
) inside of it. It does all builds under these sealed namespaces, and they have a highly restricted view of the filesystem, in fact it isn't even possible to see any other /nix/store
paths than ones you've declared as dependencies! This sealed namespace, where the build takes place, does NOT allow any other filesystems, network access, or anything else to be mounted inside the build.
If your build process says "Run this random pre-built binary we have provided, to complete the build", that binary has to work inside this environment, where it is not possible to do tricks like mounting things.
To be fair I should have been more clear about this second requirement, where the build takes place, but this same basic problem applies to NixOS too, not just the "Nix Build Environment." Any reliance on ambient paths like /usr/lib
is inevitably destined to fail and need a workaround for these cases.
It is very good for normal users though, especially on cases like macOS or Windows (anything to help reduce reliance on third party things), so I think the overall idea is a good one, even if it can't help Nix people.
from buck2.
I'll restore env vars.
from buck2.
Committed. Closing the issue. Please reopen if this does not work as expected.
from buck2.
Related Issues (20)
- Template variables for `cxx_genrule` do not work with `system_cxx_toolchain`
- Windows System Libraires HOT 4
- namespacing rules
- How to get an `artifact`'s parent directory? HOT 2
- Will Buck2 work with Sun Grid Engine for remote execution? HOT 3
- How to select cmake binary based on host platform instead of target platform?
- The "Getting Started" link is broken :-( HOT 1
- Print target when selects fail to match on toolchain targets
- Default target platform for 'buck2 run'
- Check performance of tset artifactvalues cache HOT 3
- Zsh tab completion error
- feature request - provide access to stdout for successful build actions
- rust-project: invalid method resolution for Jujutsu dep graph HOT 1
- How do I make a BXL script depend on file contents? HOT 3
- Idea: "target set" language for specifying target patterns
- Clarify expectations about external contributions
- How do I perform BXL analysis over targets with incompatible configurations? HOT 7
- feature request: support dynamic action dependencies in aquery
- BXL: format `ctx.output.print_json()` arguments
- BXL: get stdout/stderr of `actions.run()`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buck2.