First off, congrats on the Buck2 source release. I know it isn't ready yet, but I've been patiently, eagerly awaiting it for a while, and I'm so far pleased with my initial testing of it. I'm very interested in the more dynamic class of build systems that Buck2 is part of, with my previous "world champion" title holder in this category being Shake. (Also: Hi Neil!) I'm naturally already using this in fury for nothing important, which is probably a bad idea, but I hope I can give some feedback.
Here is some background. I have this file called toolchains.bzl that contains a big list of hashes, and dependencies a hash has; effectively this is just a DAG encoded as a dict. I would like to "build" each of these hashes, in practice that means doing some work to download it, then creating a symlink pointing to it (really pointing to /nix/store, but that isn't wholly relevant.)
Conceptually, the DAG of hashes is kind of like a series of "source files", where each hash is a source file, and should be "compiled" (downloaded) before any dependent "sources" are compiled. And many different targets can share these "source files". For example, here is the reverse-topologically sorted DAG for the hash 5jfg0xr0nkii0jr7v19ri9zl9fnb8cx8-rust-default-1.65.0, which you can compute yourself from the above toolchains file:
sdsqayp3k5w5hqraa3bkp1bys613q7dc-libunistring-1.0
s0w6dz5ipv87n7fn808pmzgxa4hq4bil-libidn2-2.3.2
hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163
x7h8sxz1cf5jrx1ixw5am4w300gbrjr1-cargo-1.65.0-x86_64-unknown-linux-gnu
n6mpg42fjx73y2kr1vl8ihj1ykmdhrbm-rustfmt-preview-1.65.0-x86_64-unknown-linux-gnu
nfgpn9av331q7zi1dl6d5qpir60y513s-bash-5.1-p16
k0wbm2panqbb0divlapqazbwlvcgv6m0-expand-response-params
2vqp383jfrsjb3yq0szzkirya257h1dp-gcc-11.3.0-lib
nwl7pzafadvagabksz61rg3b3cs58n9i-gmp-with-cxx-stage4-6.2.1
vv0xndc0ip83f72n0hz0wlcf3g8jhsjd-attr-2.5.1
6b882j01cn2s9xjfsxv44im4pm4b3jsr-acl-2.3.1
h48pjfgsjl75bm7f3nxcdcrqjkqwns7m-coreutils-9.1
lal84wf8mcz48srgfshj4ns1yadj1acs-zlib-1.2.13
92h8cksyz9gycda22dgbvvj2ksm01ca4-binutils-2.39
dj8gbkmgrkwndjghna8530hxavr7b5f4-linux-headers-6.0
2vbw0ga4hlxchc3hfb6443mv735h5gcp-glibc-2.35-163-bin
7p2s9z3hy317sdwfn0qc5r8qccgynlx1-glibc-2.35-163-dev
hz9w5kjpnwia847r4zvnd1dya6viqpz1-binutils-wrapper-2.39
gc7zr7wh575g1i5zs20lf3g45damwwbs-gcc-11.3.0
qga0k8h2dk8yszz1p4iz5p1awdq3ng4p-pcre-8.45
fnzj8zmxrq96vnigd0zc888qyys22jfv-gnugrep-3.7
k04h29hz6qs45pn0mzaqbyca63lrz2s0-gcc-wrapper-11.3.0
wrwx0zy8zblcsq8zwhdqbsxr2jv063fk-rustc-1.65.0-x86_64-unknown-linux-gnu
2s0sp14r5aaxhl0z16b99qcrrpfx7chi-clippy-preview-1.65.0-x86_64-unknown-linux-gnu
2dvg4fgb0lvsvr9i8qlljqj34pk2aydd-rust-docs-1.65.0-x86_64-unknown-linux-gnu
0fv17zk01p08zh6bi17m61zlfh93fcwj-rust-std-1.65.0-x86_64-unknown-linux-gnu
5jfg0xr0nkii0jr7v19ri9zl9fnb8cx8-rust-default-1.65.0
So you can read this list like so: if each line is a source input N (ranging from [0...N]), then you must build every source input [0...(N-1)] before building file N itself. Exactly what you expect.
Problem 1: anon_targets example seems broken
So two different hashes may have a common ancestor/set of dependencies, glibc is a good example because almost every hash it in its dependency tree. This seemed like a perfect use case for anonymous targets; it simply allows common work to be shared, introducing sharing that would be lost otherwise. In fact the example in that document is in some sense the same as this one; many "source files" are depended upon by multiple targets, but they don't know about the common structure between them. Therefore you can compile a single "source file" and build it once, rather than N times for each target.
But I simply can't get it to work, and because I'm new to Buck, I feel a bit lost on how to structure it. I think the problem is simply that the anon_targets function defined in there doesn't work. I have specialized it here in the below __anon_nix_targets function:
NixStoreOutputInfo = provider(fields = [ "path" ])
# this rule is run anonymously. its only job is to download a file and create a symlink to it as its sole output
def __nix_build_hash_0(ctx):
out = ctx.actions.declare_output("{}".format(ctx.attrs.hash))
storepath = "/nix/store/{}".format(ctx.attrs.hash)
ctx.actions.run(cmd_args(
["nix", "build", "--out-link", out.as_output(), storepath]
), category = "nix")
return [ DefaultInfo(default_outputs = [out]), NixStoreOutputInfo(path = out) ]
__nix_build_hash = rule(
impl = __nix_build_hash_0,
attrs = { "hash": attrs.string() },
)
# this builds many anonymous targets with the previous __nix_build_hash rule
def __anon_nix_targets(ctx, xs, k=None):
def f(hs, ps):
if len(hs) == 0:
return k(ctx, ps) if k else ps
else:
return ctx.actions.anon_target(__nix_build_hash, hs[0]).map(
lambda p: f(hs[1:], ps+[p])
)
return f(xs, [])
# this downloads a file, and symlinks it, but only after all the dependents are done
def __nix_build_toolchain_store_path_impl(ctx: "context"):
hash = "5jfg0xr0nkii0jr7v19ri9zl9fnb8cx8-rust-default-1.65.0"
deps = [
# "sdsqayp3k5w5hqraa3bkp1bys613q7dc-libunistring-1.0",
# "s0w6dz5ipv87n7fn808pmzgxa4hq4bil-libidn2-2.3.2",
# "hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163",
# "x7h8sxz1cf5jrx1ixw5am4w300gbrjr1-cargo-1.65.0-x86_64-unknown-linux-gnu",
# "n6mpg42fjx73y2kr1vl8ihj1ykmdhrbm-rustfmt-preview-1.65.0-x86_64-unknown-linux-gnu",
# "nfgpn9av331q7zi1dl6d5qpir60y513s-bash-5.1-p16",
# "k0wbm2panqbb0divlapqazbwlvcgv6m0-expand-response-params",
# "2vqp383jfrsjb3yq0szzkirya257h1dp-gcc-11.3.0-lib",
# "nwl7pzafadvagabksz61rg3b3cs58n9i-gmp-with-cxx-stage4-6.2.1",
# "vv0xndc0ip83f72n0hz0wlcf3g8jhsjd-attr-2.5.1",
# "6b882j01cn2s9xjfsxv44im4pm4b3jsr-acl-2.3.1",
# "h48pjfgsjl75bm7f3nxcdcrqjkqwns7m-coreutils-9.1",
# "lal84wf8mcz48srgfshj4ns1yadj1acs-zlib-1.2.13",
# "92h8cksyz9gycda22dgbvvj2ksm01ca4-binutils-2.39",
# "dj8gbkmgrkwndjghna8530hxavr7b5f4-linux-headers-6.0",
# "2vbw0ga4hlxchc3hfb6443mv735h5gcp-glibc-2.35-163-bin",
# "7p2s9z3hy317sdwfn0qc5r8qccgynlx1-glibc-2.35-163-dev",
# "hz9w5kjpnwia847r4zvnd1dya6viqpz1-binutils-wrapper-2.39",
# "gc7zr7wh575g1i5zs20lf3g45damwwbs-gcc-11.3.0",
# "qga0k8h2dk8yszz1p4iz5p1awdq3ng4p-pcre-8.45",
# "fnzj8zmxrq96vnigd0zc888qyys22jfv-gnugrep-3.7",
# "k04h29hz6qs45pn0mzaqbyca63lrz2s0-gcc-wrapper-11.3.0",
# "wrwx0zy8zblcsq8zwhdqbsxr2jv063fk-rustc-1.65.0-x86_64-unknown-linux-gnu",
# "2s0sp14r5aaxhl0z16b99qcrrpfx7chi-clippy-preview-1.65.0-x86_64-unknown-linux-gnu",
"2dvg4fgb0lvsvr9i8qlljqj34pk2aydd-rust-docs-1.65.0-x86_64-unknown-linux-gnu",
"0fv17zk01p08zh6bi17m61zlfh93fcwj-rust-std-1.65.0-x86_64-unknown-linux-gnu",
]
def k(ctx, ps):
deps = [p[NixStoreOutputInfo].path for p in ps]
out = ctx.actions.declare_output("{}".format(hash))
storepath = "/nix/store/{}".format(hash)
ctx.actions.run(cmd_args(
["nix", "build", "--out-link", out.as_output(), storepath]
).hidden(deps), category = "nix")
return [ DefaultInfo(default_outputs = deps + [out]), NixStoreOutputInfo(path = out) ]
return __anon_nix_targets(ctx, [{"hash": d} for d in deps], k)
__build_toolchain_store_path_rule = rule(impl = __nix_build_toolchain_store_path_impl, attrs = {})If the list deps has any number of entries > 1, then this example fails with any example TARGET:
[email protected]:~/src/buck2-nix.sl$ buck clean; buck build src/nix-depgraph:
server shutdown
/home/austin/src/buck2-nix.sl/buck-out/v2/log
/home/austin/.buck/buckd/home/austin/src/buck2-nix.sl/v2
Initialization complete, running the server.
When running analysis for `root//src/nix-depgraph:rust-stable (<unspecified>)`
Caused by:
expected a list of Provider objects, got promise()
Build ID: 5612be60-9b84-4657-97f6-64e049aedada
Jobs completed: 4. Time elapsed: 0.3s.
BUILD FAILED
However, if len(deps) == 1, i.e. you comment out the next-to-last line, then it works as expected. I think that the problem might be that if there is a single element type of type promise in a list, then Buck can figure it out. But if you have a list with many promises, it simply can't? Or something?
So I've simply been banging my head on this for a day or so, and can't find any reasonable way to make this example work that's intuitive or obvious... I actually had to fix several syntax errors in anon_targets example documentation when I took it from the example document (i.e. it uses the invalid slicing syntax xs[1...] instead of the correct xs[1:]) so I suspect it may have just been a quick example. I can send a PR to fix that, perhaps.
But it always comes back to this exact error: expected list of Providers, but got promise(). Some advice here would be nice; I feel this is close to working, though.
Problem 2: dependencies of anonymous targets need to (recursively) form a graph
The other problem in the above example, which could be handled after the previous problem, is that the dependencies don't properly form a graph structure. If we have the reverse-toposorted list:
foo
foobar
foobarbaz
foobarbazqux
Then the above example correctly specifies foobarbazqux as having all preceding entries as dependencies. But this isn't recursive: foobarbaz doesn't specify that it needs foo and foobar; foobar doesn't specify it needs foo and so on.
In my above example this isn't strictly required for correctness because the nix build command can automatically handle it. But it does mean that the graph Buck sees isn't really "complete" or accurate because it is missing the dependent structure between links.
So I don't really know the "best" way to structure this. I basically just need anonymous targets that are dependent on other anonymous targets, I guess. This is really a code structure question, and honestly I believe I'm overthinking it substantially, but I'd like some guidance to help ensure I'm doing things the "buck way" if at all possible.
It is worth noting here (or emphasizing) that the graph of dependencies above, the toolchains.bzl file, is always "statically known" up front, and is auto-generated. I could change its shape (the dict structure) as much as I want it if makes it easier, but really the list of dependencies is truly fully static, so this feels like something that should be very possible with Buck.
Other than that...
The above two problems are my current issues that I think would be amazing to solve, but I'm all ears to alternative solutions. It's possible this is an "X/Y problem" situation, I suppose.
But other than that: I'm pretty excited about Buck2 and can't wait to see it stabilize! Sorry for the long post; I realize you said "you'll probably have a bad time" using the repository currently, but you wanted feedback, and I'm happy to provide it!
Side note 1
I notice that there are no users of anon_target anywhere in the current Buck prelude, and no examples or tests of it; so perhaps something silently broke or isn't ready to be used yet? I don't know how it's used inside Meta, to be fair, so perhaps something is simply out of date.
Side note 2
I am not using the default Buck prelude, but my own prelude designed around Nix, so it would be nice if any solutions were "free standing" like the above code and didn't require the existing one.
Side note 3
The anon_targets.md documentation notes that anon_targets could be a builtin with potentially more parallelism. While that's nice, I argue something like anon_targets should really be in the Prelude itself, even if it isn't a builtin, exactly so that people aren't left to do the error prone thing of copy/pasting it everywhere like I did above, and discovered several problems.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent