Share work across configurations by setting working directory for actions then canonicalizing that working directory in RE

[thetimmorland]

Share work across configurations by setting working directory for actions then canonicalizing that working directory in RE

[cjhopman]

I think a challenge here is that basically every action has inputs that are produced by other actions. You would need to find a way to canonicalize those as well, and that's really difficult. Bazel had an interesting approach they were experimenting with described here: https://docs.google.com/document/d/17snvmic26-QdGuwVw55Gl0oOufw9sCVuOAvHqGZJFr4/edit#heading=h.5mcn15i0e1ch. I'm not sure what the status of that is today.

We're planning on experimenting with another approach for this sometime this year. There's a lot of complications (that bazel doc talks about a few and has comments on some others).

[thetimmorland]

think a challenge here is that basically every action has inputs that are produced by other actions. You would need to find a way to canonicalize those as well, and that's really difficult.

Assuming the build does not use transitions wouldn't generated inputs already be canonicalized?

PWD=buck-out/v2/gen/root/00000000000000 ./../../../../gen_src.py -o __gen_src__/generated.c
PWD=buck-out/v2/gen/root/00000000000000 cc __gen_src__/generated.c -o __main__/main

[cjhopman]

I had assumed you meant a model where the action sees its output with path 00000000000 but then it gets rewritten to the real path, because it keeping the 00000000 path just completely doesn't work. Different actions need to produce different output paths, and that needs to be some deterministic mapping in the context of any possible build.

[zjturner]

this sounds very similar to the problem I’ve discussed a bunch of times in the past, where it’s very difficult to apply different configurations per target, so you’re forced into global configuration that affects every targets hash even when it’s unnecessary.

ive had some success with transition rules to strip out unnecessary constraints, and they’ve discussed implementing a feature called “configuration trimming” to automatically strip unnecessary constraints, but no guidance yet on if or when that will actually happen

[thetimmorland]

I had assumed you meant a model where the action sees its output with path 00000000000 but then it gets rewritten to the real path, because it keeping the 00000000 path just completely doesn't work. Different actions need to produce different output paths, and that needs to be some deterministic mapping in the context of any possible build.

From buck2's perspective, each action still has a unique output path. However, by applying the transformation described above before sending an action to RE and reversing it when you receive your response (there is some bookkeeping required for this), you are able to share cache hits between configurations by hiding the configuration hash from RE.

ive had some success with transition rules to strip out unnecessary constraints, and they’ve discussed implementing a feature called “configuration trimming” to automatically strip unnecessary constraints, but no guidance yet on if or when that will actually happen

Yes, this definitely achieves a similar purpose to configuration trimming, either manual or automatic. I've read your previous threads and they've been very helpful :)