yunify/qingcloud-cloud-controller-manager

A kubernetes cloud-controller-manager for the qingcloud

qingcloud-cloud-controller-manager's Issues

Update service.loadBalancer.ingress[0].ip when change LB eip

qingcloud-cloud-controller-manager/pkg/qingcloud/loadbalancer_impl.go

Line 118 in 7b98a3c

return lb.EnsureQingCloudLB()

Investigate if LB could reuse for different services

2，每个service都会新建一个负载均衡器，多个server是否可以复用同一个负载均衡，每个负载均衡可以有10个监听器，理论上10个service可以关联至同一个负载均衡器上，只关联一个太浪费，所以是否能在service的annotations配置项里制定复用已存在的负载均衡器

service 以及 ingress的 annotation使用不方便

目前每个字段都对应一个annotation， annotation非常多， 在与ks集成的时候， 都需要用户输入很长的annotation key， 很不方便。 是否可以考虑这样， 将众多的annotation就聚合成一个json数据结构， 然后统一放到一个annotation当中。 在ks呈现的时候， 界面中最好只需要填写对应key的值，然后由前端聚合生成json， 这样使用会方便一点。

support auto ip assign

auto get an public ip if user has ip quota (Useful in test environment but in real world because public ip is costly)

Investigate if qingcloud lb could use http protocol but backend k8s svc is tcp

3，目前创建的负载均衡监听器监听协议默认是TCP，虽然不影响http请求，但监听器的监控图标里却没有请求延迟、请求次数这些项，很不方便，所以是否能在service的annotations配置项里配置负载均衡监听器的监听协议HTTP/TCP

在kubesphere中安装成功，但启动POD中报错

环境：

• 青云的机器，自己安装kubespehre，3个master, 3个worker
• kubernetes 1.22
• kubesphere 3.2.0
• 按照安装指南的步骤都做了，但启动POD后，我看到了错误：

I```
1212 15:33:56.635206 1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController

W1212 15:33:56.709736 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x120 pc=0x1974e44]

goroutine 1 [running]:

github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.(*QingCloudClient).ensureSecurityGroupByName(0xc000477080, 0x1e24d54, 0x11, 0x0, 0x0, 0xc0005ec960)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/sg.go:137 +0xe4

github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.NewQingCloudClient(0xc0005ea960, 0x1e312c9, 0x1a, 0x0, 0x0, 0x1b50760, 0xc0006cba00)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/client.go:133 +0x85c

github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud.NewQingCloud(0x206c460, 0xc00045b5d8, 0xc0004c08f8, 0x9, 0xc0004c08f0, 0x0)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud/qingcloud.go:74 +0x2dd

github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud.init.0.func1(0x206c460, 0xc00045b5d8, 0x7fff7c548a7a, 0x9, 0xc0004c0978, 0x1)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud/qingcloud.go:56 +0x35

k8s.io/cloud-provider.GetCloudProvider(0x7fff7c548a7a, 0x9, 0x206c460, 0xc00045b5d8, 0x0, 0x0, 0x0, 0x0)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/cloud-provider/plugins.go:86 +0xcf

k8s.io/cloud-provider.InitCloudProvider(0x7fff7c548a7a, 0x9, 0x7fff7c548a93, 0x1e, 0x0, 0x0, 0x0, 0x0)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/cloud-provider/plugins.go:133 +0x131

main.cloudInitializer(0xc00045b5c8, 0xc00045b5c8, 0xc0005b2228)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/cmd/main.go:73 +0x73

k8s.io/cloud-provider/app.NewCloudControllerManagerCommand.func1(0xc0005b38c0, 0xc000571a10, 0x0, 0x3, 0x0, 0x0)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/cloud-provider/app/controllermanager.go:85 +0x27f

github.com/spf13/cobra.(*Command).execute(0xc0005b38c0, 0xc0001a4050, 0x3, 0x3, 0xc0005b38c0, 0xc0001a4050)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/github.com/spf13/cobra/command.go:850 +0x472

github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b38c0, 0x1ef3fc0, 0xc000570fc0, 0x0)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/github.com/spf13/cobra/command.go:958 +0x375

github.com/spf13/cobra.(*Command).Execute(...)

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/github.com/spf13/cobra/command.go:895

main.main()

/Users/cumirror/project/go/src/github.com/yunify/qingcloud-cloud-controller-manager/cmd/main.go:64 +0x27a

自动更新LB后端

在创建一个LB svc的时候，目前是随机转发到两个node节点，如果有节点下线，那应该自动更新LB的后端

基础网络LB部署后获取的EIP有问题

ccm route controller 添加路由

可以通过ccm router controller 添加node podCIDR路由到vpc， 这样再配合flannel也可以做到k8s 集群之外直接访问podCIDR， 目前貌似青云 vpc不支持支持这样添加路由？

when changing ports in service, old listeners are not deleted

make e2e-test more robustness to detect lb deletion

http://jenkins.magicsong.xyz:8083/blue/organizations/jenkins/qingcloud-cloud-controller-manager/detail/master/1/pipeline
http://jenkins.magicsong.xyz:8083/blue/organizations/jenkins/qingcloud-cloud-controller-manager/detail/PR-54/1/pipeline

add e2e tests

servicecontroller panic

1. create cluster
2. create loadbalancer type service
3. update qingcloud-cloud-controller-manager static pod（such as: log level）, trigger recreate pod。
4. panic
5. delete all loadbalancer type service, qingcloud-cloud-controller-manager back to normal

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1338ccd714]

goroutine 39 [running]:
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x113
panic(0x133a7aaae0, 0x133c7252c0)
        /usr/lib/go/src/runtime/panic.go:491 +0x287
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/api/v1/helper.LoadBalancerStatusEqual(0xc420116440, 0x0, 0xc4206b4000)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/api/v1/helper/helpers.go:82 +0x34
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).createLoadBalancerIfNeeded(0xc4202b09a0, 0xc4206905a0, 0x11, 0xc4206b4000, 0xc4206a6cb8, 0x1338d37b94, 0xc42061c7a0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:300 +0x3e9
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processServiceUpdate(0xc4202b09a0, 0xc42027a3b0, 0xc4206b4000, 0xc4206905a0, 0x11, 0x0, 0x0, 0x0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:230 +0x73
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).syncService(0xc4202b09a0, 0xc4206905a0, 0x11, 0x0, 0x0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:748 +0x3ac
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker.func1(0xc4202b09a0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:201 +0xdb
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker(0xc4202b09a0)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:205 +0x2d
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).(github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.worker)-fm()
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:183 +0x2c
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420427c60)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x60
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420427c60, 0x3b9aca00, 0x0, 0x1, 0xc420072780)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xbf
github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420427c60, 0x3b9aca00, 0xc420072780)
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4f
created by github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).Run
        /app/src/github.com/yunify/qingcloud-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:183 +0x20a

COPY failed: no such file or directory

docker build的时候，最后一步报错提示找不到/var/lib/docker/overlay2/下面的文件？？

bind a tag to resources

we need tag for front-end

Expose more LB options through ingress annotations

Running a second layer ingress controller is annoying, would love to use qingcloud managed LB to handle the ingress traffic directly, for instance:

1. Support More listener type HTTP/HTTPS instead of TCP
2. Let qingcloud manage TLS certs for HTTPS and offload TLS

See https://cloud.google.com/kubernetes-engine/docs/concepts/ingress for some examples with GKE and GCE.

Need test in private cloud

Right now there is no private cloud for me to test

当集群规模比较大的时候， 添加后端的策略得重新考虑?

目前k8s得cloud得框架是会给EnsueLoadBalancer传递所有节点信息， 而qingcloud cloud controller manager也会全部添加这些后端， 是否可以考虑设计一些选择策略来决定是否添加后端

Using yaml as config format instead of using `gcfg`

right now, we use gcfg to configure this plugin, with more and more features added, users may need to learn how to use gcfg. It is unnecessary, just yaml

Using lb pool to speedup lb creating

As creating a lb in qingcloud is so slow now, we can create some lbs in advance

x509: failed to load system roots and no roots provided

investigate if internal LB requires security group as defualt

1，默认创建的内网负载衡器带防火墙，因为内网，防火墙不是必选项，所以是否能在service的annotations配置项里配置是否创建防火墙关联，或者默认就不要创建防火墙关联了

Avoid using host path. Use configmap or secrets

Great advice from @wnxn 👍

cloud-controller-manager pod panic after deleting a svc with loadbalancer

1. create a svc with loadbalancer type and specify qingcloud lb
2. the svc can be created and loadbalancer works fine
3. there're panic log like below in cloud-controller-manager pod after deleting the svc

I0721 05:35:03.915811       1 service_controller.go:729] Service has been deleted test-single-cluster/nginx-single-cluster. Attempting to cleanup load balancer resources
I0721 05:35:03.916266       1 event.go:258] Event(v1.ObjectReference{Kind:"Service", Namespace:"test-single-cluster", Name:"nginx-single-cluster", UID:"85a52f9e-94e8-4fcd-8da2-94a9b70a4ca6", APIVersion:"v1", ResourceVersion:"1689842", FieldPath:""}): type: 'Normal' reason: 'DeletingLoadBalancer' Deleting load balancer
I0721 05:35:26.795646       1 loadbalancer_impl.go:139] DeleteLoadBalancer takes total 22 seconds
E0721 05:35:26.795775       1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 242 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15d4360, 0x25b5890)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:69 +0x7b
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:51 +0x82
panic(0x15d4360, 0x25b5890)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/yunify/qingcloud-sdk-go/service.(*TagService).DetachTags(0x0, 0xc0002c7b40, 0x0, 0x1, 0x0)
	/Users/zheng1/work/go/pkg/mod/github.com/yunify/[email protected]/service/tag.go:272 +0x4e
github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.DetachTagsFromResources(0x0, 0x0, 0x0, 0x0, 0xc000625ac0, 0x1, 0x1, 0x17eee03, 0x3, 0x0, ...)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/tags.go:43 +0x12f
github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.(*qingCloudLoadBalanceExecutor).Delete(0xc000014230, 0xc0005dbdd0, 0xb, 0x0, 0xc)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/lb.go:305 +0x4b8
github.com/yunify/qingcloud-cloud-controller-manager/pkg/loadbalance.(*LoadBalancer).DeleteQingCloudLB(0xc0009b1380, 0x1a67540, 0xc000046048)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/loadbalance/loadbalancer.go:452 +0xf9
github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud.(*QingCloud).EnsureLoadBalancerDeleted(0xc000c02270, 0x1a67540, 0xc000046048, 0x17f68e5, 0xa, 0xc000441000, 0x0, 0x0)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud/loadbalancer_impl.go:142 +0xdb
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processLoadBalancerDelete(0xc000616620, 0xc00000e4f0, 0xc000864810, 0x28, 0x1, 0x2d9)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:760 +0x141
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processServiceDeletion(0xc000616620, 0xc000864810, 0x28, 0x4a, 0xc000bd5d48)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:750 +0x7a
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).syncService(0xc000616620, 0xc000864810, 0x28, 0x0, 0x0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:730 +0x40a
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processNextWorkItem(0xc000616620, 0x1a07a00)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:213 +0xf0
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker(0xc000616620)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:202 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00041bcc0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00041bcc0, 0x3b9aca00, 0x0, 0x1, 0xc00008e720)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc00041bcc0, 0x3b9aca00, 0xc00008e720)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by k8s.io/kubernetes/pkg/controller/service.(*ServiceController).Run
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:191 +0x205
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7c176e]
goroutine 242 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:58 +0x105
panic(0x15d4360, 0x25b5890)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/yunify/qingcloud-sdk-go/service.(*TagService).DetachTags(0x0, 0xc0002c7b40, 0x0, 0x1, 0x0)
	/Users/zheng1/work/go/pkg/mod/github.com/yunify/[email protected]/service/tag.go:272 +0x4e
github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.DetachTagsFromResources(0x0, 0x0, 0x0, 0x0, 0xc000625ac0, 0x1, 0x1, 0x17eee03, 0x3, 0x0, ...)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/tags.go:43 +0x12f
github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor.(*qingCloudLoadBalanceExecutor).Delete(0xc000014230, 0xc0005dbdd0, 0xb, 0x0, 0xc)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/executor/lb.go:305 +0x4b8
github.com/yunify/qingcloud-cloud-controller-manager/pkg/loadbalance.(*LoadBalancer).DeleteQingCloudLB(0xc0009b1380, 0x1a67540, 0xc000046048)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/loadbalance/loadbalancer.go:452 +0xf9
github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud.(*QingCloud).EnsureLoadBalancerDeleted(0xc000c02270, 0x1a67540, 0xc000046048, 0x17f68e5, 0xa, 0xc000441000, 0x0, 0x0)
	/Users/zheng1/work/go/src/github.com/yunify/qingcloud-cloud-controller-manager/pkg/qingcloud/loadbalancer_impl.go:142 +0xdb
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processLoadBalancerDelete(0xc000616620, 0xc00000e4f0, 0xc000864810, 0x28, 0x1, 0x2d9)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:760 +0x141
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processServiceDeletion(0xc000616620, 0xc000864810, 0x28, 0x4a, 0xc000bd5d48)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:750 +0x7a
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).syncService(0xc000616620, 0xc000864810, 0x28, 0x0, 0x0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:730 +0x40a
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processNextWorkItem(0xc000616620, 0x1a07a00)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:213 +0xf0
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker(0xc000616620)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:202 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00041bcc0)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00041bcc0, 0x3b9aca00, 0x0, 0x1, 0xc00008e720)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc00041bcc0, 0x3b9aca00, 0xc00008e720)
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by k8s.io/kubernetes/pkg/controller/service.(*ServiceController).Run
	/Users/zheng1/work/go/pkg/mod/k8s.io/[email protected]/pkg/controller/service/service_controller.go:191 +0x

docs about new config

/kind documentation

Allow users specify nodes count for service LB

QingCloud SDK params:

https://github.com/yunify/qingcloud-sdk-go/blob/c7eee83bd5e8238d767995d84b20109d34405eec/service/load_balancer.go#L360

Refactor error handling.

right now, our error handling looks like a chaos

LoadBalancer 改进

• 用户不需要设置任何 annotation 就可以创建 LoadBalancer，默认创建私网的 LoadBalancer(或者公网？主要考虑到国内公网 ip 的成本比较高).
• 私网的 LoadBalancer 不需要用户指定 vxnet，默认使用当前集群所在的 vxnet，通过 cloud config 指定默认的 vxnet。

support using existing lb

/kind feature

Enhancement LoadBalancer EnsureLoadBalancer strategy

1. Compare new loadbalancer spec with qingcloud loadbalancer's attributes, and find diff attributes。
2. Only update the necessary attributes。

Support hostnic

normally the backend of lb is nodeport, but after we use hostnic, we can set the backend pod ip directly.

update docs

Update docs to help user install cloud manager

Missing cloud provider configuration file

There should be a cloud provider configuration file，while runnig qingcloud-cloud-controller-manager.exe with --cloud-provider option to finish the initialization of the plugin.

But i did not find such a cloud provider configuration file in this project.

add CI framework

修改服务的lb配置方式后，未解绑服务之前绑定的lb

操作步骤：
1、在ks中已创建了一个配置了lb的服务，lb的配置为service.beta.kubernetes.io/qingcloud-load-balancer-eip-source: auto

2、待成功创建lb并绑定了lb后，修改该服务的lb配置方式为如下：
service.beta.kubernetes.io/qingcloud-load-balancer-eip-strategy为reuse-lb，
设置service.beta.kubernetes.io/qingcloud-load-balancer-id为已存在的其他的lb

3、待修改生效后，查看步骤1中服务绑定的lb下面的监听器及后端

预期结果：
修改服务的lb配置后，会解绑之前绑定的lb

实际结果：
修改服务的lb配置后，未解绑之前绑定的lb，该服务同时被绑定了2个lb

Add some UT

The app has been rewritten for extensibility. Many codes need UT for guaranteeing stable。

support creating multi node lb

wait for yunify/qingcloud-sdk-go#143

Loadbalancer backend support attach service endpoint address directly

Kubernetes LoadBalancer backend is service NodePort currently, there are some shortcomings:

1. Performance waste
2. Service get client real source ip is complicated

So, attach service endpoint address to Loadbalancer backend directly when use hostnic network.

The following improvements are required to achieve this goal：

1. Watch service endpoint change by kubernetes api.
2. Update loadbalancer backend when service endpoint change.

Should provide a config option to control this feature.

删掉annotations，没有触发删除LB监听

先删除service得注解，再修改type。没有触发LB监听器更新。