Comments (7)
@ianzhang366 I like the measurement plan you outlined here. Is the baseline going to be the performance of the old synchronizer? Also, how difficult would it be to increase the concurrency? For example, 10, 20, 30, 50, 100 concurrent subscriptions.
from multicloud-operators-subscription.
@ianzhang366 I like the plan but I feel like we should focus on correctness and throughput over performance for now.
from multicloud-operators-subscription.
@mikeshng @xiangjingli @rokej @jnpacker
Hi team, here's a quick update for this issue,
Object
- Build up the benchmark for synchronizer 1.0(batch apply model) and 2.0(the channel version)
- Compare the above 2 models and identify the bottleneck
Benchmark
-
Test method and measurement
-
Test method
Given hub and spoke clusterkubeconfig
, then apply N subscriptions on the hub, then measure the resource created time on spoke.
The subscription is pointing to a namespace channel. The namespace channel has 1 deployable which has aconfigmap
as payload. -
Test ENV
a. How to run the controller
Both tests are running against the binary file of the subscription controller on my Mac.
Sample running command:
go run cmd/manager/main.go --alsologtostderr true --hub-cluster-configfile $GOPATH/src/github.com/openshift-install-mac/hubconfig/kubeconfig --kubeconfig $GOPATH/src/github.com/openshift-install-mac/managedconfig/kubeconfig --cluster-name ocp-managed --cluster-namespace ocp-managed --disable-tls true 2> out.log
b. How to apply and measure the resource
I wrote the following tool to implement the test method. You can clone and
build the tool.https://github.com/open-cluster-management/acmapp
Sample running command:
Apply 30 subscriptions and wait till all of them created at managed cluster.acmapp perf --n 30 --hubcfg $GOPATH/src/github.com/open-cluster-management/acmapp/hub --spokecfg $GOPATH/src/github.com/open-cluster-management/acmapp/managecfg
- Measurement
Then we measure how long will it take for theconfigmap
created on spoke
cluster.
Note: we apply the N subscriptions on the hub at once.
-
-
Result
- Batch model(sync_interval is default 1 min)
- N = 10 (need 2mins)
- N = 30 (crash)
- Channel model
- N = 10 (20 seconds)
- N = 30 (around 3mins)
- N = 100 (around 27 mins)
- Batch model(sync_interval is default 1 min)
-
Bottleneck
-
Batch model
As the error shows(in raw data), when we are having large amount resource applied, the
synchronizer crashed due to the map write and read at the same time. -
Channel model
As you may notice, the channel model won't crash when N = 30, it took about 3 mins to deploy allconfigmap
on the managed cluster.However, it will take quite a long time to apply this resource, when N = 100, in the beginning, it took more than 1 hour to apply all
the resource. So I did some parameter tunning around the channel model,
mainly,
(tunning PR is:
#198
)- let the processing request waiting longer before retrying.
- when re-queuing, we try to re-queue request in a uniform manner
After these changes, when N = 100, we are able to get all the resource
deployable around 22 mins.
Moving forward, I inspected the processing time for each request. I mean, from
the synchronizer get the request to the synchronizer get back to the
request with the result.From the observed data, we can see:
- the single processing time increased when the amount of request increased
- the default '/' request kicks in too often and used up too much resource
-
-
Raw data
https://github.com/open-cluster-management/acmapp/blob/master/raw_data/benchmark_may_15th.md
from multicloud-operators-subscription.
FYI, we can use the sync.Map
instead of the map
to achieve the thread-safe map. With this, we can increase the number of workers on the cache server-side.
-
sync.Map
https://golang.org/pkg/sync/#Map -
current cache map definition
https://github.com/open-cluster-management/multicloud-operators-subscription/blob/2159f30ae0e7e77c01b80814c98c1c1c1cda666a/pkg/synchronizer/kubernetes/sync_server.go#L39-L52
from multicloud-operators-subscription.
@ianzhang366 The sync.Map doc says The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.. Do you think the synchronizer's usage is one of these?
from multicloud-operators-subscription.
I think we fall into the 2nd case since our resource map is mapping the GVK, and I assume the concurrent subscriptions are not always trying to access the same resource(GVK).
However, I'm not sure would the assumption holds true when we are talking about config map or secret... since these are very popular resources.
In addition, there're some options other than the sync.Map
, for example, the following,
https://github.com/orcaman/concurrent-map
from multicloud-operators-subscription.
moving to backlog issue to track this item
from multicloud-operators-subscription.
Related Issues (20)
- 😿 Failed to sync the upstream PRs: #371
- 😿 Failed to sync the upstream PRs: #372, #373, #375, #377, #376, #379
- 😿 Failed to sync the upstream PRs: #378
- 😿 Failed to sync the upstream PRs: #381
- 😿 Failed to sync the upstream PRs: #382 HOT 1
- 😿 Failed to sync the upstream PRs: #384
- 😿 Failed to sync the upstream PRs: #389 HOT 2
- 😿 Failed to sync the upstream PRs: #388
- 😿 Failed to sync the upstream PRs: #391
- 😿 Failed to sync the upstream PRs: #393 HOT 1
- 😿 Failed to sync the upstream PRs: #394
- 😿 Failed to sync the upstream PRs: #395 HOT 1
- 😿 Failed to sync the upstream PRs: #396
- 😿 Failed to sync the upstream PRs: #397
- 😿 Failed to sync the upstream PRs: #399
- 😿 Failed to sync the upstream PRs: #401
- 😿 Failed to sync the upstream PRs: #402
- 😿 Failed to sync the upstream PRs: #404
- 😿 Failed to sync the upstream PRs: #405 HOT 1
- Dependency Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from multicloud-operators-subscription.