Need to have performance test in place

[stolostron]

Need to have performance test in place

[rokej]

@ianzhang366 I like the measurement plan you outlined here. Is the baseline going to be the performance of the old synchronizer? Also, how difficult would it be to increase the concurrency? For example, 10, 20, 30, 50, 100 concurrent subscriptions.

[mikeshng]

@ianzhang366 I like the plan but I feel like we should focus on correctness and throughput over performance for now.

[ianzhang366]

@mikeshng @xiangjingli @rokej @jnpacker

Hi team, here's a quick update for this issue,

Object

• Build up the benchmark for synchronizer 1.0(batch apply model) and 2.0(the channel version)
• Compare the above 2 models and identify the bottleneck

Benchmark

1. Test method and measurement

   • Test method
     Given hub and spoke cluster kubeconfig, then apply N subscriptions on the hub, then measure the resource created time on spoke.
     The subscription is pointing to a namespace channel. The namespace channel has 1 deployable which has a configmap as payload.

   • Test ENV
     a. How to run the controller
     Both tests are running against the binary file of the subscription controller on my Mac.

   Sample running command:

   go run cmd/manager/main.go --alsologtostderr true --hub-cluster-configfile $GOPATH/src/github.com/openshift-install-mac/hubconfig/kubeconfig --kubeconfig $GOPATH/src/github.com/openshift-install-mac/managedconfig/kubeconfig --cluster-name ocp-managed --cluster-namespace ocp-managed  --disable-tls true  2> out.log

   b. How to apply and measure the resource
   I wrote the following tool to implement the test method. You can clone and
   build the tool.

   https://github.com/open-cluster-management/acmapp

   Sample running command:
   Apply 30 subscriptions and wait till all of them created at managed cluster.

   acmapp perf --n 30 --hubcfg $GOPATH/src/github.com/open-cluster-management/acmapp/hub --spokecfg $GOPATH/src/github.com/open-cluster-management/acmapp/managecfg

   • Measurement
     Then we measure how long will it take for the configmap created on spoke
     cluster.

   Note: we apply the N subscriptions on the hub at once.

2. Result

   • Batch model(sync_interval is default 1 min)
     • N = 10 (need 2mins)
     • N = 30 (crash)
   • Channel model
     • N = 10 (20 seconds)
     • N = 30 (around 3mins)
     • N = 100 (around 27 mins)

3. Bottleneck

   • Batch model
     As the error shows(in raw data), when we are having large amount resource applied, the
     synchronizer crashed due to the map write and read at the same time.

   • Channel model
     As you may notice, the channel model won't crash when N = 30, it took about 3 mins to deploy all configmap on the managed cluster.

     However, it will take quite a long time to apply this resource, when N = 100, in the beginning, it took more than 1 hour to apply all
     the resource. So I did some parameter tunning around the channel model,
     mainly,
     (tunning PR is:
     #198
     )

     1. let the processing request waiting longer before retrying.
     2. when re-queuing, we try to re-queue request in a uniform manner
        After these changes, when N = 100, we are able to get all the resource
        deployable around 22 mins.

     Moving forward, I inspected the processing time for each request. I mean, from
     the synchronizer get the request to the synchronizer get back to the
     request with the result.

     From the observed data, we can see:

     1. the single processing time increased when the amount of request increased
     2. the default '/' request kicks in too often and used up too much resource

4. Raw data
   https://github.com/open-cluster-management/acmapp/blob/master/raw_data/benchmark_may_15th.md

[ianzhang366]

FYI, we can use the sync.Map instead of the map to achieve the thread-safe map. With this, we can increase the number of workers on the cache server-side.

• sync.Map
  https://golang.org/pkg/sync/#Map

• current cache map definition
  https://github.com/open-cluster-management/multicloud-operators-subscription/blob/2159f30ae0e7e77c01b80814c98c1c1c1cda666a/pkg/synchronizer/kubernetes/sync_server.go#L39-L52

[rokej]

@ianzhang366 The sync.Map doc says The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.. Do you think the synchronizer's usage is one of these?

[ianzhang366]

I think we fall into the 2nd case since our resource map is mapping the GVK, and I assume the concurrent subscriptions are not always trying to access the same resource(GVK).

However, I'm not sure would the assumption holds true when we are talking about config map or secret... since these are very popular resources.

In addition, there're some options other than the sync.Map, for example, the following,
https://github.com/orcaman/concurrent-map

[ianzhang366]

moving to backlog issue to track this item