Comments (13)
@xbmono yes, we can fix this by changing the cache key to consider the full path of the workspace.
from broccoli-persistent-filter.
We use monorepo for our ember addons and apps and we use lerna
to manage it. Until recently we were running commands in series but we wanted to make running tests faster so we experimented with lerna's parallelism but after few attempts we started having the error described here.
@xbmono yes, we can fix this by changing the cache key to consider the full path of the workspace.
I think such a fix would be good enough while keeping the advantage of persistent cache.
Would you reconsider your approach? I think there are few use-cases which would benefit from it even thought it was not originally meant to be used for concurrent runs.
Or if those are use-case you want to avoid, could you please consider adding some option which could be provided over ember-cli-build.js
file instead of environment variable?
from broccoli-persistent-filter.
One other relevant note is that I realized that in my case it's also possible for the exact same app [w/ same deps] to be building multiple times in parallel in different folders. Basically if the same commit ends up on multiple branches that are being tracked by CI, it could end up building them both at the same time.
from broccoli-persistent-filter.
Thank you for reporting! I believe we were aware (generally speaking) that a race could occur here. We should try add in some more resiliency to ensure that cross-process caching does not fail the build. I believe that in the worst case scenario, we should just opt out of caching and recalculate.
@stefanpenner - Thoughts?
from broccoli-persistent-filter.
building the same project (exact some on-disk files/deps etc) more then once concurrently isn't really something we support. At-least not with a persistent cache... It is possible to support this, but It would come at either a complexity and/or performance cost to the regular case.
It might be a good idea to have your CI box limit its concurrency somewhat, or concurrent tasks (like building) will starve each other for resources...
I don't think this is an issue with this library, but we should make clear reference to it in the readme.
If we want to mitigate this issue, we may want to an a locking mechanism to ember-cli itself, essentially disallowing the same project to be actively build concurrently.
@workmanw your above "Work-around" actually seems like the exact "configuration" I would expect for supporting concurrent builds. Alternatively, we could also enable an "opt-out" of persistent cache altogether... as you are then essentially opting out of persistent-cache entirely on your CI box.
from broccoli-persistent-filter.
@stefanpenner I'm 👍 for making it an option. I know most people run their builds in a containerized service (Travis, etc) and that makes us in the minority here.
building the same project (exact some on-disk files/deps etc)
To be clear, it's the same project/deps but on a different branch in a different cloned instance of the repo (and thus in a different directory). A single GitHub post hook can trigger both builds, so they end up running simultaneously and thus fight over the system temp cache.
I do think we would get some value out of having a configuration for the cache path because in our case, we build both dev and prod versions of the app back to back. So without really knowing too many of the details, it sounds like there would be some cache benefit for that.
I'm fine to continue using BROCCOLI_PERSISTENT_FILTER_CACHE_ROOT
. But it would be nice to have it documented and marked public so I don't have to worry about regressions. And I wouldn't mind a slightly shorter ENV variable 😃 .
from broccoli-persistent-filter.
To be clear, it's the same project/deps but on a different branch in a different cloned instance of the repo (and thus in a different directory). A single GitHub post hook can trigger both builds, so they end up running simultaneously and thus fight over the system temp cache.
Something seems fishy then, because the cached files should have different mtime/atimes and then ultimately not collide...
I'm assuming you come to this issue via BroccoliPersistentFilter (or one of its many subclasses?) If so, then maybe we are missing including something important in the cache key..
from broccoli-persistent-filter.
@stefanpenner One point of interest is that it only ever seemed to happen on prod builds (w/ fingerprinting). I don't know if that's relevant or not.
I'm happy to help in any way I can. We could do a screen hero if you want. But honestly I don't want to suck up your time if I'm the only one experiencing this. And the debugging is not super pleasant because it's all over SSH. That said, I'm always happy to help.
from broccoli-persistent-filter.
when I run unit tests in my Ember app in parallel, even if they are in separate workspace so no shared folder, I get this error, I wonder if this is related:
[emberTests2] cleaning up...
[emberTests2] Build failed.
[emberTests2] Build Canceled: Broccoli Builder ran into an error with `Babel` plugin. 💥
[emberTests2] unexpected end of file
[emberTests2] Error: unexpected end of file
[emberTests2] at Zlib._handle.onerror (zlib.js:370:17)
from broccoli-persistent-filter.
@stefanpenner thanks it worked :)
from broccoli-persistent-filter.
Or if those are use-case you want to avoid, could you please consider adding some option which could be provided over ember-cli-build.js file instead of environment variable?
Environment variables can easily be set in ember-cli-build.js
:
process.env.SOME_AWESOME_THING = "salsa";
@xbmono yes, we can fix this by changing the cache key to consider the full path of the workspace.
I think such a fix would be good enough while keeping the advantage of persistent cache.
Would you reconsider your approach?
I'm not sure how to interpret this. Are you saying that @stefanpenner's suggestion would work or would not work? As far as I can tell his suggestion would solve the specific problem reported (amongst others), and we would gladly review/accept pull requests to implement. What do we need to reconsider?
from broccoli-persistent-filter.
Environment variables can easily be set in ember-cli-build.js
Thanks for the tip, I wasn't aware of this method.
I'm not sure how to interpret this. Are you saying that @stefanpenner's suggestion would work or would not work?
I think that if the full path is part of the cache key then there won't be any race condition which means it would fix both CI use-case and monorepo commands running in parallel.
I wanted you to reconsider the method of the caching mechanism.
from broccoli-persistent-filter.
@stefanpenner I just hit this issue when trying to build a single app on Windows using WSL 2. The same workaround applies.
from broccoli-persistent-filter.
Related Issues (20)
- Version 10 of node.js has been released
- Changelog is outdated HOT 1
- Add typescript types
- broccoli-persistent-filter - childprocess fail HOT 13
- An in-range update of mocha is breaking the build 🚨 HOT 5
- Prevent inconsistent ember-cli builds with different extensions HOT 7
- cacheKeyProcessString does not invalidate across rebuilds. HOT 1
- Worker is terminated HOT 4
- Rejecting / Excluding Files | broccoli-funnel HOT 5
- Use glob expression as file dependency HOT 1
- Upgrade promise-map-series to async/await version once ready
- move to broccoli org? HOT 3
- `dependencyInvalidation` and `concurrency` should not be required HOT 2
- Typescript issues with 3.0 release. HOT 3
- Node 10 doesn't support recursive directory operations. HOT 3
- Build Error (OneShot) HOT 1
- Switch to GH Actions
- broccoli-persistent-filter:babel random error in windows HOT 2
- "unexpected end of file" can persist from corrupted build and there is no discoverable way to force rebuild
- Bring back Windows CI...
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from broccoli-persistent-filter.