kurator-dev/kurator

Unified resource orchestration, unified scheduling, unified traffic management and unified telemetry for distributed cloud

Kurator

Overview

Kurator is an open source distributed cloud native platform that helps users to build their own distributed cloud native infrastructure and facilitates enterprise digital transformation.

Kurator stands on shoulders of many popular cloud native software stacks including Kubernetes, Istio, Prometheus, FluxCD, KubeEdge, Volcano, Karmada, Kyverno, etc. It provides powerful capabilities for multi-cloud and multi-cluster management, including:

• Multi-cloud, Edge-cloud, Edge-edge Synergy
• Unified Resource Orchestration
• Unified Scheduling
• Unified Traffic Management
• Unified Telemetry

Advantages

• Infrastructure-as-Code: declarative way of infrastructure(cluster, node, vpc, etc) management on cloud, edge or on premises.
• Out of box: one button to install cloud native software stacks
• Unified management of clusters with fleet:

1. Support cluster registration and un-registration to a fleet.
2. Application customize and sync across fleet.
3. Namespaces, ServiceAccount, Service sameness across clusters of a fleet.
4. Provide service discovery and communication across clusters.
5. Aggregate metrics from all clusters of a fleet.
6. Provide policy engine to make all clusters have consistent policies.

Architecture

Documentation

Please visit kurator website for our documentation.

Contact

If you have any question, feel free to reach out to us in the following ways:

• mailing group
• slack

Contributing

If you're interested in being a contributor and want to get involved in developing the Kurator code, please see CONTRIBUTING for details on submitting patches and the contribution workflow.

License

Kurator is under the Apache 2.0 license. See the LICENSE file for details.

report a vulnerability

If you find a vulnerability in Kurator, you can report it to our security-team in the following way. We will deal with it as soon as possible.

kurator's Issues

Support flat network between multiclusters

What would you like to be added:

There are multi solutions to make the connectivity between different clusters.
Currently we can make use of istio to do that.

There are many other light weight solutions like submariner and NSM, both have their use cases.

I think we should make use of them to achieve flat network between clusters.

Why is this needed:

[cluster operator] deleting cluster requires further more manual deletion besides “kubectl delete cluster”

What would you like to be added:
After execting kubectl delete cluster ${CLUSTER_NAME}
Resources cluster/awscluster will get stuck due to finalizer.
Resource awsclusterstaticidentity will remain.
Why is this needed:
further explanation on procedures of deletion.

Change docker image hub to `docker.io/kuratordev`

Have created an account in dockerhub, since kurator was used by other org. We named kuratordev, which aligns with kurator domain.

Why do not register kurator-dev?

Because docker hub only supports letters and digitals

Add doc for prometheus federation

Kurator has supported Prometheus federation, but we still need docs about how that works

support pixie as part of observability

plan to add pixie as new observability plugin:

• support install pixie vizier
• support install pixier cloud

cc @hzxuzhonghu @prodanlabs any further idea?

support more complex network mode in istio

In #46 we add non-flat mode about istio, but in a real world the typology of clusters may more complex.
thinking about a network typology like following:

we need a mechanism to support this but can be used in other plugin.

cc @hzxuzhonghu @prodanlabs

kurator install supports specifying component configuration files

Please provide an in-depth description of the question you have:
The components component configuration file of kurator is embedded in the binary file. If the user needs to modify the components component configuration file, it will be very inconvenient.

What do you think about this question?:
Can kurator install add a flag to use the local component configuration file.I don't know if this is possible.

Environment:

• Kurator version:
• Others:

improve istio non flat

enchant istio non flat implement after karmada-io/karmada#2236 released, maybe in v1.3?

[FR] implement infra cluster

• API #134
• Controller
  • #146
  • #160
  • IRSA #185
• Webhook #172
• Docs

Incorrect tools list command output

What happened:

The tool list command will show all components configuration plus Status

the output maybe as below we can see the status looks super confusing.

kurator tool list -ojson
{
	"argocd": {
		"Cli": "",
		"Hub": "",
		"Name": "argocd",
		"ReleaseURLPrefix": "https://github.com/argoproj/argo-cd/releases/download/",
		"Status": "NotReady",
		"Version": "v2.4.8"
	},
	"istio": {
		"Cli": "",
		"Hub": "docker.io/istio",
		"Name": "istio",
		"ReleaseURLPrefix": "https://github.com/istio/istio/releases/download",
		"Status": "NotReady",
		"Version": "1.13.3"
	},
	"karmada": {
		"Cli": "",
		"Hub": "",
		"Name": "karmada",
		"ReleaseURLPrefix": "https://github.com/karmada-io/karmada/releases/download",
		"Status": "NotReady",
		"Version": "v1.2.1"
	},
	"kubeedge": {
		"Cli": "",
		"Hub": "docker.io/kebeedge",
		"Name": "kubeedge",
		"ReleaseURLPrefix": "https://github.com/kubeedge/kubeedge/releases/download",
		"Status": "NotReady",
		"Version": "1.10.0"
	},
	"submariner": {
		"Cli": "",
		"Hub": "",
		"Name": "submariner",
		"ReleaseURLPrefix": "https://github.com/submariner-io/releases/releases/download",
		"Status": "NotReady",
		"Version": "v0.12.2"
	}
}

What you expected to happen:

I would expect the status to indicate whether the component is installed, whether it is healthy.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kurator version:
• Others:

Support for install kubernetes

Please provide an in-depth description of the question you have:

All infrastructure or kurator functions are implemented on the basis of kubernetes, because we think that supporting the installation of kubernetes can be put on the agenda.

What do you think about this question?:
I personally prefer to use kubeadm to install, you can use ClusterConfiguration to define many installation parameters, such as

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  serviceSubnet: "10.254.0.0/16"
  podSubnet: "192.168.0.0/24"
  dnsDomain: "cluster.local"
controlPlaneEndpoint: "172.31.6.161:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  certSANs:
  - "172.31.6.161"
  timeoutForControlPlane: 4m0s
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
clusterName: "foo-cluster-01"

Environment:

• Kurator version:
• Others:

improve Istio network mode

address comment #81 (comment)
after #81 , we can customize cluster network when installing istio
we should remove --network-mode, let user define cluster network with labels on Cluster CR.

istio install eastwest gateway

Please provide an in-depth description of the question you have:
First the eastwest gateway makes sense to me.
I executed kurator install istio --primary dev-cluster-01 --remote dev-cluster-02, and eastwest gateway was not installed.

What do you think about this question?:

I just want, is it possible to install eastwest gateway by default.

Environment:

• Kurator version:
• Others:

Local up setup cluster should contains node other than master

Currently，we setup local cluster by kind, and by default it create a cluster with only master node.

In some case, it need to install components like submariner gateway engineer in normal node.

[CustomCluster] Investigating kubespray to optimize the process of VMs cluster management

• Can and how to let kurator get the reason why the kubespray command fails to execute
• How to let kurator get the kubeconfig of the installed cluster on VMs. finish at ##279
• Can and how to reduce cluster installation time on VMs, based on the current fact that it takes at least 15 minutes to install after the kubespray image is pulled down

Why is this needed:
Cluster management on VMs realized by customCluster controller relies on kubespray, investigation of kubespray will help to optimize the process of vms cluster management

cc @hzxuzhonghu @zirain

add log output

Please provide an in-depth description of the question you have:

In the case of a bad network, downloading the tool will be very slow, I was stuck for about 3 minutes. It can be confusing if there is no log output, i don't know if kurator is working.

What do you think about this question?:
Print a log here telling the user that the xxx tool is being downloaded.
https://github.com/kurator-dev/kurator/blob/main/pkg/util/download.go#L35

Environment:

• Kurator version:
• Others:

make `go get kurator.dev/kurator` work

replace kurator.dev/kurator => github.com/kurator-dev/kurator v0.0.0-20230209094936-fe8627f10e44

require (
	kurator.dev/kurator v0.0.0-20230209094936-fe8627f10e44
)

cc @hzxuzhonghu

Increase golangci-lint timeout

I have met https://github.com/kurator-dev/kurator/runs/6959249310?check_suite_focus=true

level=error msg="Running error: context loading failed: failed to load packages: timed out to load packages: context deadline exceeded"
level=error msg="Timeout exceeded: try increasing it by passing --timeout option"
make: *** [golangci-lint] Error 1

cc @gy95

Cluster lifecycle management

Kurator should support cluster lifecycle(creation, upgrade, scale) management.

First need to support k8s cluster creation on following cases:

1. existing infra, existing nodes
2. cloud provider with infra(node, vpc, volume, etc) creation on the fly

Remove `--feature-gates` from manifest

It was removed In #117. I believe it is useless left in manifest

crd and code gen should be included in `make gen`

Supplement customcluster controller test

refactor components.yaml

What would you like to be added:

split components.yaml to two parts:

• tools.yaml describe how to get binary like helm, karmadactl etc
• components.yaml(rename to plugin.yaml) describe to relationship between component and tool chart and other things

cc @hzxuzhonghu

Consistent policy enforcement

Add document directory

Please provide an in-depth description of the question you have:

I'm researching kurator and kurator looks like a good fit for us. I can output some installation or other documentation during the research process.

I don't know if the community has a documentation plan.

What do you think about this question?:

I want to put the documentation in the docs directory.

Environment:

• Kurator version:
• Others:

support running on non root user

cluster-operator need to support `cluster-operator version`

Custom cluster high availablity deploy

Investigating support GraphQL

What would you like to be added:

FYI https://graphql.org/

GraphQL is a query language for your API, and a server-side runtime for executing queries using a type system you define for your data.

The benifit is Get many resources in a single request

As how to support GraphQL, i donot have a clear thought right now. Any comments are welcomed.

Why is this needed:

Currently, if we want to check a workload and its correslated resources, we must do many queries. With graphQL supported, this can be largely saved.

custommachine controller improve

https://github.com/kurator-dev/kurator/blob/main/pkg/controllers/custommachine_controller.go#L73

Now withi the above call, it always fail. And this controller falls into a dead loop

We should check calling of UpdateReferenceAPIContract

Verify go mod

We should verify go mod and go sum with CI

Support cluster resource dump

Further more, provide a tool to migrate cluster. This will include works as below:

1. dump cluster resources
2. image migration
3. resource requests analysis

[Umbrella] Add `tool` subcommand

Please provide an in-depth description of the question you have:

According to the issues discussion, add the tool subcommand to output information such as the version of tools, and download tools.

task:

• add tool list subcommand , output toos information.
• add tool install subcommand, download tools.

correct worker count

"msg"="Starting workers" "controller"="customcluster" "controllerGroup"="cluster.kurator.dev" "controllerKind"="CustomCluster" "worker count"=1

Support continuous delivery

What would you like to be added:

Flux and ArgoCD are both right GitOps tool for Kubernetes cluster, and are both cncf incubating projects.

There are many comparations between them.

https://thenewstack.io/gitops-on-kubernetes-deciding-between-argo-cd-and-flux/

https://rajputvaibhav.medium.com/argo-cd-vs-flux-cd-right-gitops-tool-for-your-kubernetes-cluster-c71cff489d26#:~:text=Argo%20lets%20you%20connect%20multiple,application%20and%20provide%20different%20manifests.

What they do seems similar, but ArgoCD supports rich UI, so I suggest kurator support ArgoCD first.

Section<Install cluster operator>lacks infos on <your_custom_values>.yaml

What would you like to be added:

In https://github.com/kurator-dev/kurator/blob/main/docs/content/en/docs/Cluster%20operator/quickstart.md#install-cluster-operator,<your_custom_values>.yaml comes out of nowhere
Why is this needed:
,confused about file "<your_custom_values>.yaml". Cannot find the purpose of this file and what inside it.
@zirain

Revisit --tempDir

What happened:

From #22 (comment), I observed that temp-dir has a default value, which seems not correct, as this is inconsistent.

kurator install -h

      --temp-dir string          file path including temporary generated files (default "/tmp/kurator1559865940")

After revisiting this flag, it is only used to store temporarily generated yaml by istio. I am thinking can we remove this flag, and keep the temp files in istio install dir.

cc @zirain

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kurator version:
• Others:

Enhancement of CustomCluster

• support custom registry of kubespray
• support other kubernetes configs
• support accessing k8s api from outside of the provisioned cluster
• #152

cc @hzxuzhonghu @Xieql

Are there any plans to support KubeVela

Please provide an in-depth description of the question you have:

What do you think about this question?:

Environment:

• Kurator version:
• Others:

can you provide more details on how to apply aws credentials in aws cluster operator

What would you like to be added:
actions on how to apply aws credentials in aws cluster operator

Why is this needed:
stuck on creating aws machine due to lack of credentials

Improve logging

when debuging argoCD integration. I see logsm, which is duplicate and bad formatted.

DEBU[2022-08-11 11:35:46] components: map[argocd:{Name:argocd Version:v2.4.8 Hub: ReleaseURLPrefix:https://github.com/argoproj/argo-cd/releases/download/} istio:{Name:istio Version:1.13.3 Hub:docker.io/istio ReleaseURLPrefix:https://github.com/istio/istio/releases/download} karmada:{Name:karmada Version:v1.2.1 Hub: ReleaseURLPrefix:https://github.com/karmada-io/karmada/releases/download} kubeedge:{Name:kubeedge Version:1.10.0 Hub:docker.io/kebeedge ReleaseURLPrefix:https://github.com/kubeedge/kubeedge/releases/download} prometheus:{Name:prometheus Version: Hub: ReleaseURLPrefix:} submariner:{Name:submariner Version:v0.12.2 Hub: ReleaseURLPrefix:https://github.com/submariner-io/releases/releases/download} volcano:{Name:volcano Version:1.5.1 Hub:docker.io/volcanosh ReleaseURLPrefix:https://raw.githubusercontent.com/volcano-sh/volcano/}] 
DEBU[2022-08-11 11:35:46] components: map[argocd:{Name:argocd Version:v2.4.8 Hub: ReleaseURLPrefix:https://github.com/argoproj/argo-cd/releases/download/} istio:{Name:istio Version:1.13.3 Hub:docker.io/istio ReleaseURLPrefix:https://github.com/istio/istio/releases/download} karmada:{Name:karmada Version:v1.2.1 Hub: ReleaseURLPrefix:https://github.com/karmada-io/karmada/releases/download} kubeedge:{Name:kubeedge Version:1.10.0 Hub:docker.io/kebeedge ReleaseURLPrefix:https://github.com/kubeedge/kubeedge/releases/download} prometheus:{Name:prometheus Version: Hub: ReleaseURLPrefix:} submariner:{Name:submariner Version:v0.12.2 Hub: ReleaseURLPrefix:https://github.com/submariner-io/releases/releases/download} volcano:{Name:volcano Version:1.5.1 Hub:docker.io/volcanosh ReleaseURLPrefix:https://raw.githubusercontent.com/volcano-sh/volcano/}] 
DEBU[2022-08-11 11:35:46] start install argoCD: &{DryRun:false config:0xc0001a8b00 Ui:0xc00097a6c0 HomeDir:/root/.kurator WaitInterval:1s WaitTimeout:2m0s KubeConfig:/root/.kube/kurator-host.config KubeContext: Components:map[argocd:{Name:argocd Version:v2.4.8 Hub: ReleaseURLPrefix:https://github.com/argoproj/argo-cd/releases/download/} istio:{Name:istio Version:1.13.3 Hub:docker.io/istio ReleaseURLPrefix:https://github.com/istio/istio/releases/download} karmada:{Name:karmada Version:v1.2.1 Hub: ReleaseURLPrefix:https://github.com/karmada-io/karmada/releases/download} kubeedge:{Name:kubeedge Version:1.10.0 Hub:docker.io/kebeedge ReleaseURLPrefix:https://github.com/kubeedge/kubeedge/releases/download} prometheus:{Name:prometheus Version: Hub: ReleaseURLPrefix:} submariner:{Name:submariner Version:v0.12.2 Hub: ReleaseURLPrefix:https://github.com/submariner-io/releases/releases/download} volcano:{Name:volcano Version:1.5.1 Hub:docker.io/volcanosh ReleaseURLPrefix:https://raw.githubusercontent.com/volcano-sh/volcano/}]}  

kurator install should be idempotent

I have met after install some component with kurator failed the first time, it always fails afterwards.
One cause I have found is that some resources can be created at the first time, but the second time it still create same resources. That can happen on helm client now.

Not sure if there are other places that is not idempotent, need to investigate.

kurator looks like a one-stop platform

Please provide an in-depth description of the question you have:

Does kurator just install cloud-native application services? Is there anything else planned.

I am interested in kurator.

What do you think about this question?:

Environment:

• Kurator version:
• Others:

kurator install volcano failed

kurator install volcano
INFO[2022-08-26 14:56:22] begin to download resource https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/volcano-development.yaml ->  
ERRO[2022-08-26 14:56:22] volcano execute error: received 404 status code from https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/volcano-development.yaml 
Error: volcano execute error: received 404 status code from https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/volcano-development.yaml
execute kurator command failed:  volcano execute error: received 404 status code from https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/volcano-development.yaml

make gen-check failed

What happened:
make gen-check failed

# make gen-check 
go mod tidy -compat=1.17
go clean -testcache
go clean -cache
rm -rf out/linux-amd64
rm -rf out/prom
go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
go install -a github.com/brancz/gojsontoyaml@latest
go install -a github.com/google/go-jsonnet/cmd/jsonnet@latest
rm -rf manifests/profiles/prom/
mkdir -p manifests/profiles/prom/
mkdir -p out/prom
cp manifests/jsonnet/kube-prometheus.jsonnet out/prom
hack/gen-prom.sh out/prom v0.10.0 kube-prometheus.jsonnet
begin to generate prom manifests
path: out/prom
version: v0.10.0
jsonnet file: kube-prometheus.jsonnet
~/go/src/github.com/kurator-dev/kurator/out/prom ~/go/src/github.com/kurator-dev/kurator
hack/gen-prom.sh: line 13: jb: command not found
Makefile:65: recipe for target 'gen-prom' failed
make: *** [gen-prom] Error 127

Solutions:
we should export PATH=$PATH:$GOPATH/bin after we install jb

kurator/Makefile

Lines 57 to 61 in 33fef92

	.PHONY: install-tools
	install-tools:
	go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
	go install -a github.com/brancz/gojsontoyaml@latest
	go install -a github.com/google/go-jsonnet/cmd/jsonnet@latest

or we can use the function defined in util.sh

kurator/hack/util.sh

Lines 13 to 20 in 33fef92

	function util::install_tools() {
	local package="$1"
	local version="$2"
	echo "go install ${package}@${version}"
	GO111MODULE=on go install "${package}"@"${version}"
	GOPATH=$(go env GOPATH | awk -F ':' '{print $1}')
	export PATH=$PATH:$GOPATH/bin
	}

but i don't know how to run functions of shell script in Makefile

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kurator version:
• Others:

Can a service `istiod` of type NodePort be supported

Please provide an in-depth description of the question you have:
I'm not sure if istiod necessarily needs a service of type LoadBalancer.
https://github.com/kurator-dev/kurator/blob/main/pkg/plugin/istio/install.go#L315-L347

There is another question, why is ingress.Hostname judged here
https://github.com/kurator-dev/kurator/blob/main/pkg/plugin/istio/install.go#L462

I use Cloud LoadBalancer, but the LoadBalancer Ingress Hostname of istiod-elb is empty. I'm a little confused.

What do you think about this question?:

Environment:

• Kurator version:
• Others:

move gen component out of makefile

address comment from #77 (comment)

add `config` subcommand

Today we discussed kurator internally,the tool subcommand doesn't make much sense. For this function, it's my fault,i didn't think clearly.

By default, the kurator configuration file is embedded in the binary file. If you want to view the configuration, you need to go to the kurator repository to view the configuration file,if you want to modify the configuration file (such as changing the component version), you still need to go to the kurator repository to download the configuration file to the current server, and then modify it. It is inconvenient for users.

So what we need is the config subcommand, not the tool, our idea is

• kurator tool list changed to kurator config list/show . View version information for all components.
• Add kurator config gen/dump command to generate default configuration file.

@hzxuzhonghu @zirain What do you think?

Remove dependency on cert-manager to provide certificate for webhook server

Support mount self managed certificate

Provide production prometheus federation and long term storage capability

What would you like to be added:

Previously we support prometheus federation, which is a basic model that can do multicluster metrics aggregate, but far from production ready.

I have investigated recently, and this one https://elastisys.com/long-term-metrics-storage-thanos-vs-cortex-vs-influxdb-vs-m3db-vs-victoriametrics-vs-timescaledb/ is the most complete comparison with clear criteria.

The conclusion is thanos is the winner for long time storage of multicluster metrics.

As we can see thanos some some dependencies https://thanos.io/tip/thanos/getting-started.md/#dependencies Particularly we should pay attention to object storage, which we have not supported in kurator.

@daixiang0, for the s3 object storage, can we use minio to provide rather than using AWS directly?