Comments (7)
Do you have a test project where I can reproduce this? Without a repro there's not much I can do.
from clj-kondo.
from clj-kondo.
I guess you can run the modified version locally and see if you can find out more.
from clj-kondo.
So I added some logging to clj-kondo.impl.cache/with-cache
and saw that pool-1-thread-N
threads (N is 1..9 on my computer) compete for the cache lock. They make 1323 attempts to lock the file and 117 times these fail. Out of those failures 102 are the 0th retry (i. e. first try), 12 - at 1st retry, 2 - at 2nd retry, 1 - at 3rd retry:
retry# - count
0 - 102
1 - 12
2 - 2
3 - 1
I suspect that the number of parallel threads (as when run on a powerful CI host) creates increased contention which results in reaching the max retry attempts for some threads. My logging shows that the max number of retries is set to 6. So on my computer with 9 parallel threads it seems unlikely (although possible) to run into this issue. The number of processors is predefined in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/core.clj#L364 so there isn't much I can currently do to change this. The number of retries is predefined in code in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/analyzer/namespace.clj#L290 and in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/cache.clj#L179.
I tried running locally with increased number of threads. With 26 threads, it failed to obtain lock 291 times and the retry counts were also higher:
retry# - count
0 - 194
1 - 64
2 - 17
3 - 8
4 - 4
It seems to me that to resolve the issue we should make either the number of retries higher and/or configurable or the number of parallel threads lower and/or configurable.
Thoughts?
from clj-kondo.
I don't know the exact answer of what tuning will be best, but maybe we could make threads configurable at least.
from clj-kondo.
I don't know the exact answer of what tuning will be best, but maybe we could make threads configurable at least.
I'd say we never want this to fail, so we probably should increase the number of retries. We could also potentially create a queue of cache updates so that they're guaranteed to execute in the order they happen and no thread is delayed too much.
from clj-kondo.
I like the queue idea
from clj-kondo.
Related Issues (20)
- clj-kondo Pod - lint stdin HOT 1
- clj-kondo incorrectly flags method as field HOT 1
- Wrong unresolved error on defmacro HOT 1
- false positive(?) : Static fields should be referenced without parens unless they are intended as function calls HOT 6
- :case-symbol-test should warn on symbol in list
- :java-static-field-call isn't documented HOT 1
- Warn on actual vs expected order in tests HOT 4
- Array class notation will change to`String/1`, per CLJ-2833 HOT 2
- Pre/Post maps break unused-value and misplaced-docstring lints
- Import configurations to intermediate directory instead of top level
- Useless for expression not reported as an unused value
- Linter: unresolved type hint class names HOT 3
- Custom file types HOT 1
- False positive unused JS library HOT 1
- Linting ignore for Class/forName
- Support linting JavaScript vars in clj-kondo
- Support linting JavaScript vars in clj-kondo HOT 1
- The `:redefined-var` documentation is outdated HOT 2
- Location missing on literal syntax errors HOT 4
- Detect redundant calls to str HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clj-kondo.