Comments (3)
There are a number of possible causes for increased RAM usage.
What is your constraints and constraint template count? (you mention 87 for constraints, but not clear if/how that relates to # of constraint templates)
Has Gatekeeper been configured to cap the number of concurrent inbound requests? (if not, then each concurrent request may require more RAM)
Are you using referential data? If so has the data set grown? (referential data is cached in RAM)
To get a rough sense of whether RAM usage is due to at-rest conditions vs. satisfying serving requirements, I suggest disabling calling of validating/mutating webhooks temporarily to see what the RAM usage settles on.
If at-rest usage is still high, likely culprit is referential data.
If request volume is high, then I'd suspect either uncapped parallelism or poorly-optimized Rego in a constraint template.
Another option might be to compare webhook memory usage to audit pod memory usage. If Audit looks healthier, likely it's due to QPS.
The error you are highlighting is consistent with golang context being closed due to OOMKill (I.e. OOMKill causes the error, not the other way around).
from gatekeeper.
The constraint template count is the same as constraints.
We are using referential data and removing pods from it made things better for both audit and the webhook; they were both consuming high memory.
I also noticed that it ended up needing more resources in a smaller multi-tenant cluster than another larger cluster so I'd still like to cap the number of requests, how can I do that?
Also despite it not being OOMKilled anymore I am continuing to see the error I mentioned above, even though the webhook continues to work as expected end to end.
from gatekeeper.
"context canceled" can mean the caller's context was canceled (usually due to request timeout). I think it usually says something different than "serving context canceled", though maybe the framework changed its error text.
In any case, --max-serving-threads
is the flag you want for capping # of simultaneous calls to Rego, though it usually defaults to GOMAXPROCS:
gatekeeper/pkg/webhook/policy.go
Line 71 in 80d677a
Tuning that value and/or raising CPU or # of serving webhook pods may help with timeouts (assuming that's what the context canceled errors are)
from gatekeeper.
Related Issues (20)
- No warning displayed on kubectl apply with enforcementAction warn HOT 4
- Course aggregation of request duration metrics HOT 2
- Add: app.kubernetes.io/name label to the Deployment object HOT 2
- Migrate psp Templates. HOT 2
- Add a flag for GK validating webhook to defer to vap
- admission webhook "validation.gatekeeper.sh" denied the request HOT 2
- publish images with microarch levels HOT 1
- gatekeeper-controller logs do not display HOT 1
- Metric names mismatch: `*_count` in document, `*_count_total` in actual behavior HOT 1
- doc: Add a page to include all flag information in one place HOT 1
- New example for location value when using complex Labels
- 404 Helm chart repo not found HOT 4
- Pass additional info in the mutation request to external data provider HOT 1
- Interpolation in mutation hooks for namespace or other parameters HOT 1
- Upgrade Gatekeeper to use Debian 12 Distroless HOT 3
- WebhookConfigurations(mutating and Validating) causing slow pod creation
- Can't use Gator cli to verify opa with external_data
- Change chart to only set matchConditions on webhooks when the value parameter is not empty HOT 1
- Support `--log-stats-audit` / `--log-stats-admission` in Helm chart HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gatekeeper.