Comments (8)
Yes, this is expected behavior. The only caching that is available across daemon instances is whatever is provided by the RE action cache.
We've considered improvements here, typically under names like "dice rehydration" or something like that, which would preserve daemon state across restarts. Nothing that's on any near future roadmap though, unfortunately
from buck2.
to be clear, is there no concept of a local action cache?
from buck2.
There is not, no, although this is an interesting alternative to dice rehydration that I hadn't considered. I think @scottcao knows some things about this, maybe he can weigh in
from buck2.
I was under the impression that hash_all_commands basically enabled the action cache and that it would have benefits even in local build scenarios. Did I misunderstand?
from buck2.
I was under the impression that hash_all_commands basically enabled the action cache and that it would have benefits even in local build scenarios.
It does, but not across daemons.
Buck2 always has at least two strategies by which it avoids unnecessarily rerunning actions. The first is a remote action cache, which isn't relevant here. The second is DICE. DICE backs the implementation of most of buck2 core and is the reason that running the same command twice doesn't cause it to do all the work twice - it prevents not just unnecessarily rerunning actions, but also unnecessarily reevaluating buck files and things like that.
Dice, however, is not a cache, but rather an incrementality engine. It's primarily concerned with detecting what did or didn't change since the last build. As a result, if an action was already run five commands ago but invalidated since then, dice won't detect that it can skip recomputing that. Similarly, if two actions "swap places" in the build graph, dice won't detect to not rerun them either.
My understanding of hash_all_commands
is that it activates a local action cache that can additionally avoid redoing work because of the above restrictions. However, that cache is still limited to the lifetime of a single daemon
from buck2.
Just curious, what does DICE stand for? I keep seeing it and wondering.
In any case, I know buck1 had a dir cache, and the topic of dir cache has come up in the context of buck2 several times in the past. What's the current status of that? It seems like, if dir cache were implemented in buck1, would that also include action cache? And have there been any updates about implementing it on Meta's side?
Hypothetically speaking, let's say someone outside of Meta wanted to implement dir cache and upstream the work. Is it feasible without having access to internal Meta systems?
from buck2.
Just curious, what does DICE stand for? I keep seeing it and wondering.
Distributed incremental computation engine. The "distributed" is ehhh... "future work"
What's the current status of that? It seems like, if dir cache were implemented in buck1, would that also include action cache?
I don't actually know what the dir cache in buck1 did, so I may have to get back to you on that.
Internally, I don't think we currently have any concrete plans for additional local caches - remote caches are working well for us.
Hypothetically speaking, let's say someone outside of Meta wanted to implement dir cache and upstream the work. Is it feasible without having access to internal Meta systems?
Yes, certainly; most of the internal-only code is around RE and this should basically work separately. We'd also be very happy to generally review design proposals or anything like that.
But even if that weren't the case - if there is some change that generally makes sense for OSS but we have no use for internally, feel free to send a PR anyway, I have no problem #[cfg]
ing it out in our builds
from buck2.
Related Issues (20)
- Golden testing? HOT 4
- Watchman/buck2 integration HOT 2
- How can you get a library's output path from bxl? HOT 7
- Why does bxl_actions require an execution platform? HOT 1
- Remote execution can't find executable HOT 13
- haskell_library fails on MacOS
- Feature Request: Shutdown buck2 server after some idle time
- --show-output (and related options show nothing in column 2) when using DefaultInfo with default_outs HOT 1
- Suggestion: Use tar or zip for release artifacts
- Why can not Buck trigger Buildbarn's g++ using system_cxx_toolchain?
- 'buck2 build //tests:' can not find 'sh' when using case 'buck2/examples/remote_execution/buildbarn/tests'
- License details (for SPDX compliance) HOT 2
- Buck2 performance on single-file update in erlang project HOT 13
- `SRCDIR` is not absolute in genrule script HOT 2
- It should be possible to consume multiple `default_outputs` in an `attrs.list(attrs.source())` HOT 5
- RE: upload cancelled with "stream error: stream no longer needed" HOT 5
- Zig support HOT 2
- Handling environment variables for local and remote builds HOT 1
- How can I check and/or override the path for the linker for rust/cxx? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buck2.