kubecube-io / kubecube Goto Github PK

View Code? Open in Web Editor NEW

483.0 10.0 67.0 23.5 MB

KubeCube is an open source enterprise-level container platform

Home Page: https://kubecube.io

License: Apache License 2.0

Makefile 0.52% Dockerfile 0.28% Go 98.28% Shell 0.92%

kubernetes cloud-native observability multi-tenancy multi-clusters arm-support certified-kubernetes

kubecube's People

Stargazers

Watchers

Forkers

isgasho mingningshao windbise hadoop835 jangocheng jangocity fish-pro qianyanjishu wittech mgbin088 weilaaa hanhongyuan githubstack wzqnls snailtolight johsonluo eagle9527 xiaochong547 sunhaokim zhengyansheng linkinghack lgh990 puluto geek3carl bitini111 cyofeiyue aiical ozzpy fusida tiancandevloper mfuuzy jamesyang1986 cjysmat chuabo acproject ilibx hzliangbin showsmall drcayman git2017-jpg liuflylove666 jimzhu0804 eryeru12 snpyeso yourkeychen zccdy rebortboss goodpaas limengcanyu wrhb123 nnuujj sandaman2015 iqiuyu-0821 fengshunli caoyingjunz lbemi yupengzhangl iq-scm superspeedone keonjeo mamh-mixed wangguangxin2023 djkko cf1998 guoguodan aragorn1026 j4ckzh0u

kubecube's Issues

centos7.4安装失败

centos7.4, all in one安装脚本时报错，拉不到镜像

2021-07-13 14:27:52 DEBUG enable and start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /etc/systemd/system/docker.service.
2021-07-13 14:27:57 INFO downloading images
I0713 14:27:59.066899   15693 version.go:252] remote version is much newer: v1.21.2; falling back to: stable-1.19
W0713 14:27:59.839835   15693 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]

 2021-07-13 14:27:59 DEBUG spin pid: 15728                                                                                                                                                                -Error response from daemon: Get https://registry.cn-hangzhou.aliyuncs.com/v2/google_containers/kube-apiserver/manifests/v1.19.12: Get https://dockerauth.cn-hangzhou.aliyuncs.com/auth?scope=repository%3Agoogle_containers%2Fkube-apiserver%3Apull&service=registry.aliyuncs.com%3Acn-hangzhou%3A26842: dial tcp: lookup dockerauth.cn-hangzhou.aliyuncs.com on 10.198.141.241:53: no answer from DNS server
2021-07-13 14:32:41 ERROR install kubernetes failed

Helm安装kubecube问题

pivotKubeConfig 和localKubeConfig需要怎么配置求教

[Bug] helm安装时 Nodeport 和已经存在的Ingress的port冲突且无法规避

Error: INSTALLATION FAILED: failed to create resource: Service "frontend-nodeport" is invalid: spec.ports[0].nodePort: Invalid value: 30080: provided port is already allocated

helm安装时必须填写NodePort的服务器所在的IP, 但是安装过程会和集群已经存在的ingress的NodePort冲突. 且无法规避这个端口, 导致无法安装.

上面的30080端口就是Ingress的NodePort.

[Feature]能否增加离线安装包

太多情况下会是内网环境，或者有一台可以上网，其他服务起运行安装脚本太程序有点繁琐和复杂

安装过程应该支持为warden服务的init-container设置http proxy

当部署环境为需要代理连接互联网时，无法完成helm charts下载。

可以考虑将helm-charts包直接打包在容器镜像中，或者支持为init container配置可选的http_proxy | https_proxy环境变量。

[Bug]按照文档一路安装，从1.2-到1.4【logseer】真用不起

继续观光一下
【Hotplug】hotplugs.hotplug.kubecube.io v1
common 与 pivot-cluster 都打开

spec:
  component:
    -
      name: audit
      status: enabled
    -
      env: "address: elasticsearch-master-headless.elasticsearch.svc\n"
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: enabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: enabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://172.31.0.171:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: enabled

spec:
  component:
    -
      env: "address: elasticsearch-master.elasticsearch.svc \n"
      name: logseer
      status: enabled
    -
      env: "grafana:\n  enabled: true\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://kubecube-thanos-receive:19291/api/v1/receive\n"
      name: kubecube-monitoring
    -
      env: "receive:\n  tsdbRetention: 7d\n  replicaCount: 1\n  replicationFactor: 1\n"
      name: kubecube-thanos
      status: enabled

默认配置，配置 elasticsearch-master-headless.elasticsearch.svc和elasticsearch-master.elasticsearch.svc 都配置过，理论上不会有什么影响，还是不行,然后进行调试

问题一：查询日志报错 “request elasticsearch fail”
问题二：操作审计无数据（经过调试已解决）
过程如下：
查看logseer运行pod的容器日志发现如下

2022-09-24 20:40:47.299 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpress"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "elasticsearch-master.elasticsearch.svc:9200"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1664019350313,
                "lte" : 1664022950313,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1m",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-09-24 20:40:48.302 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
	at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

出现一个空指针异常

要么取出ES的数据为空
要么查找拼接的地址为空毕竟有一个没看到访问HOST地址的请求 request to es, url: /*/_search?ignore_unavailable=true
进入logseer容器

直接curl http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true 发现返回了一大堆数据，证明ES连通性是好的，不过毕竟没添加参数，curl不是很好加参数，加了参数可能就返回空报错了。
猜测是不是环境变量没有设置起，不停的调整环境变量格式，甚至configMap,内部配置文件期望出现日志
equest to es, url: http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true。
是不是没有读取到address: elasticsearch-master.elasticsearch.svc 这个变量，最终放弃也许日志本来就是这么写的。
转入logagent 的filebeat 的configMap 发现
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:30435]
这个根本访问不到修改成
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:9200]
再试试，嗯一样的不通(好在的是filebeate不爆连接错误了 )
接着看了一下文档也没发现哪里不对，再修复下审计
我本来也安装的内部ES，还是当外部配置下吧

kubectl edit deploy audit -n kubecube-system
env:
- name: AUDIT_WEBHOOK_HOST
  value: http://elasticsearch-master.elasticsearch:9200
- name: AUDIT_WEBHOOK_INDEX
  value: audit
- name: AUDIT_WEBHOOK_TYPE
  value: logs

审计可以了，

但是日志依然不通，看来只有想办法开放ES 9200端口出来用工具连连是没上传还是没查询到，
不过大体定位到如下可能的几个问题

可能是logseer没有读取到环境变量导致出错
filebeate配置错误没有上传成功，导致查询空数据（这个数据内容空就报错应该不至于），但是确实直接安装的filebeat有一个connect 错误需要修复
费解

目前发现的问题猜测ripple和filebeat的配置感觉这里嫌疑最大，创建了新的日志抓取任务，也没看到/etc/filebeat/inputs.d 有什么文件改动
不过也建议修复下空报错的问题，让指示得更明显，只能去看看哪里有源码了

[Feature] worker cluster shouldn't share same kubeconfig with pivot cluster

Warden should only have the basic authority for updating and watch resource's change in pivot. Otherwise owner of warden cluster could operate the pivot cluster. It has a security risky of kubecube

[Feature] non-k8s resource authz expansion

Why do we need non-k8s resource authz expansion?

Assumed that we have a bookinfo server and we want to make a decision who can access bookinfo and by how?

How to expand authz of non-k8s resource?

Introduce new crd ExternalResource for mapping non-k8s resource so that we can use it as general k8s resource to rbac

apiVersion: extension.kubecube.io/v1
kind: ExternalResource
metadata:
  name: bookinfo
spec:
  namespaced: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform-admin
rules:
  ...
  - apiGroups:
      - "extension.kubecube.io/v1"
    resources:
      - externalresources
    resourceNames:
      - bookinfo
    verbs:
      - get
      - list
      - watch
      - create
      - delete
      - deletecollection
      - patch
      - update

[Bug] All-in-one 安装文档执行错误

执行按All-in-one安装文档执行有如下错误：

使用文档地址：https://www.kubecube.io/docs/installation-guide/all-in-one/#%E5%9C%A8-linux-%E4%B8%8A%E9%83%A8%E7%BD%B2-kubecube

[Feature]KubeCube适配接入用户自有监控后端

Is your feature request related to a problem? Please describe.

目前用户可以修改hotplug关掉监控，但是关闭后重新登录显示401；�
用户无法接入自有监控后端。

Describe the solution you'd like

修改hotplug，不允许关闭监控功能；
或修改代码后添加文档，指导用户接入自有监控后端。

Describe alternatives you've considered

Additional context

[Bug] 部署脚本缺少必要的终止判断

kubecube 部署的时候，有不少过程是上一步运行失败，但是不中断，继续往下执行，导致后面失败的问题。。

类似helm 执行失败后，应该终止shell，而不是进入下一步
https://github.com/kubecube-io/kubecube-installer/blob/main/install_kubecube.sh#L121

[Bug] When add cluster, If my kubeconfig is wrong. kubecube container will panic.

Describe the bug
A clear and concise description of what the bug is.
可以使用中文。
construct a kind cluster without sign outside ip.
when visit this kind cluster will get error Get "https://192.168.4.124:57300/api?timeout=32s": x509: certificate is valid for 10.96.0.1, 172.18.0.3, not xxxx
Of course, this is a problem with my configuration, but the kubecube program should not panic

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Feature] 组件监控里面为什么没有信息

我看我的k8s集群里面都有这些pod，很奇怪，求大神指导

[Bug] Deployments 部署时环境变量无法输入下划线

Deployments 部署，环境变量填写的时候，输入下划线报错。系统提示是可以输入下划线的，但是输入下划线报错，应该是判断出现了问题，如下图。

[Feature]官网提供的安装教程以及其他信息很久没更新

使用官网提供的helm安装方法部署，能部署的最新版本是1.8.2，大概是23年4月的版本，但github的版本已经到1.9.0
如果已安装旧版本，有升级到1.9.0的流程吗
官网的说明资料有些是21年不知是否是最新
期待更新
另外，产品有提供helm管理的模块吗，包括helm repo，通过helm安装的应用管理。有些中间件部署如果是helm方式安装，如何在平台管理控制？
在租户功能界面，只有创建项目，没有删除项目的功能，项目改名但不能更改项目标识

[Bug]跟ceph存储集群混合部署会无法正常运行kubecube

Describe the bug
A clear and concise description of what the bug is.
可以使用中文。
场景一：
先决条件：
三台服务器搭建3节点，k8s 已部署v1.27.6，k8s已通过helm方式安装 metrics-server
同时三节点通过cephadm方式部署ceph存储集群，正常运行

操作
通过helm 方式安装kubecube
1、发现kubecube-monitoring-prometheus-node-exporter-XXX
9100端口与ceph自带节点监控服务冲突无法启动
通过修改ceph默认端口，让kubecube可以使用9100，解决

2、发现prometheus-kubecube-monitoring-prometheus-0
服务没有启动，也没拉取镜像等动作，监控服务相关如下

helm list -A
查询发现部分服务安装失败
kubecube-monitoring kubecube-monitoring 1 2024-02-29 08:48:09.902374875 +0000 UTC failed kubecube-monitoring-15.4.12 0.47.0
kubecube-thanos kubecube-monitoring 1 2024-02-29 08:48:58.754995455 +0000 UTC failed thanos-3.18.0 0.21.1

场景二
基于场景一无法正常使用kubecube情况，调整如下
三台服务器搭建3节点，k8s 已部署v1.27.6，k8s已通过helm方式安装 metrics-server
cephadm集群卸载不启动

操作
1、通过helm 方式安装kubecube ，kubecube正常安装所有组件，并正常运行
2、使用cephadm安装ceph存储集群，初始化集群，并修改ceph自带节点监控服务端口为9111避免冲突。观察kubecube正常运行
3、使用ceph对每一台机进行存储硬盘初始化，启动OSD服务。此时对应主机上kubecube相关的pod出现crashoff/error，崩溃并且无法自动重启，而在这台机的其他pod，例如kube-proxy，kube-controller-manager等是正常的。
4、针对第3步，将问题服务器节点重启，则kubecube相关的pod，恢复正常，而ceph服务也是正常，OSD服务正常。
5、三个节点均是这种情况，ceph执行初始化OSD服务，对应的主机上的kubecube的pod就崩溃无法自动重启。然后整台服务器重启后，kubecube恢复正常，ceph正常，运行1天，暂时没有异常。

server(please complete the following information):

OS: kylinV10
k8s version：v1.27.6
kubecube version：v1.8.2

[Bug] I installed all in one,bug there's some problerm with me

Internal error occurred: failed calling webhook "vresourcequota.kb.io": failed to call webhook: Post "https://warden.kubecube-system.svc:8443/validate-core-kubernetes-v1-resource-quota?timeout=10s": service "warden" not found

[Feature] Integration with olm app market

Problem
The problem that Integration with olm app market is authR, users in KubeCube need have related rbac of new operator crds to do access.

Solution may be
olm OperatorGroup records all of gvks during specified namespace, we can use a controller to watch for OperatorGroup and create Role and RoleBinding for user according to OperatorGroup info

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operators.coreos.com/v1","kind":"OperatorGroup","metadata":{"annotations":{},"name":"og-single","namespace":"default"},"spec":{"targetNamespaces":["default"]}}
    olm.providedAPIs: EtcdBackup.v1beta2.etcd.database.coreos.com,EtcdCluster.v1beta2.etcd.database.coreos.com,EtcdRestore.v1beta2.etcd.database.coreos.com
  creationTimestamp: "2021-09-28T09:27:20Z"
  generation: 1
  name: og-single
  namespace: default
  resourceVersion: "17745074"
  uid: 23d1f838-f3df-4025-ac82-51fa69212606
spec:
  targetNamespaces:
  - default
status:
  lastUpdated: "2021-09-28T09:27:20Z"
  namespaces:
  - default

Meanwhile the controller should aggregate the ClusterRole of OperatorGroup to platform-admin so that he/she can access new crds of operator

aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      olm.opgroup.permissions/aggregate-to-c571d720f17289d3-admin: "true"
  - matchLabels:
      olm.opgroup.permissions/aggregate-to-2c1e6f7e17c07035-admin: "true"
  - matchLabels:
      olm.opgroup.permissions/aggregate-to-2fdc3540750c4d2b-admin: "true"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2021-09-28T09:27:20Z"
  labels:
    olm.owner: og-single
    olm.owner.kind: OperatorGroup
    olm.owner.namespace: default
    rbac.authorization.k8s.io/aggregate-to-platform-admin: "true"
  name: og-single-admin
  resourceVersion: "20351534"
  uid: 3e1689c1-272e-4d36-b3c4-310ac7dbf884
rules:
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdclusters
  verbs:
  - '*'
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdbackups
  verbs:
  - '*'
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdrestores
  verbs:
  - '*'

[Feature] 文档中建议添加如何移除kubecube相关文档

目前使用中发现有如下问题：
1、文档中只有安装介绍，没有如何移除kubecube的介绍
2、在已有kubernetes集群中安装的文档中，应该补充说明如何预先使用可插拔忽略某些组件的安装，比如ingress-nginx以及monintor等，避免在不知情情况下与集群中已经安装的产生冲突

[Feature]适应企业组织的多级租户模型的项目与空间概念问题

从企业组织关系角度来看，常见地，项目是一个企业分配公司资源的最小粒度单元
在Kubcube的组织管理中，创建租户，租户可绑定（N用户，N项目）

在Kubcube的资源管理中，租户分配租用了资源，下一步分配租户资源。按逻辑来，理应继承组织管理的层级，将租户资源分配给租户内的多个项目

但是，此时，就多了一个“空间”和“创建空间分配资源”的概念，让人觉得很奇怪。因为在一个企业组织中，很少有说我在一个空间中，基本上是我在哪个项目中。企业内带头人成立项目，下一步是项目审批通过了，就下发公司资源。好像很少说项目审批通过了，再创造一个空间，创建多个空间来下发公司资源。

请问在资源管理时，为什么不把租户资源直接按项目来分配？这样的企业组织逻辑不是更清晰吗？空间和创建空间的最佳实践案例是什么？

创建项目时报错

创建项目时出现如下错误，请问从哪能解决

[Feature]hnc support hncresourcequota，是不是可以使用最新的来更新了？

Is your feature request related to a problem? Please describe.
最新的hnc 1.1版本，已经支持hncresourcequota了，是否可以后续支持hncresourcequota来控制配额了。

Describe alternatives you've considered
最新的hnc 1.1版本，已经支持hncresourcequota了，是否可以后续支持hncresourcequota来控制配额了。

kubecube all in one 安装方式的一些疑问

all in one 的安装方式是不是会自动创建一个默认的k8s集群

[Bug]日志查询报错，开启logersee后 request elasticsearch fail

使用日志采集

apiVersion: netease.com/v1
kind: Logconfig
metadata:
  creationTimestamp: '2022-06-13T05:59:27Z'
  generation: 1
  labels:
    app: dep-staff
  managedFields:
    -
      apiVersion: netease.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app': {}
        'f:spec':
          .: {}
          'f:inputs': {}
      manager: Mozilla
      operation: Update
      time: '2022-06-13T05:59:27Z'
  name: stufflog
  namespace: cqdx
  resourceVersion: '50486'
  uid: 4c2ba645-762e-4354-bae4-d9cddd1e18b4
spec:
  inputs:
    -
      enable: true
      type:
        name: dockerStdout

点击日志查询

报错的接口为

检查热插拔组件

[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME            PHASE   AGE
common          fail    2d2h
pivot-cluster   fail    2d2h

查看组件

spec:
  component:
    -
      name: audit
      status: disabled
    -
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: disabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: disabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://10.10.10.44:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: disabled
status:
  phase: fail
  results:
    -
      message: 'audit is disabled'
      name: audit
      result: success
      status: disabled
    -
      message: uninstalled
      name: logseer
      result: success
      status: disabled
    -
      message: 'release is running'
      name: logagent
      result: success
      status: enabled
    -
      message: 'release is running'
      name: elasticsearch
      result: success
      status: enabled
    -
      message: 'helm install fail, cannot re-use a name that is still in use'
      name: kubecube-monitoring
      result: fail
      status: enabled
    -
      message: 'release is running'
      name: kubecube-thanos
      result: success
      status: enabled

看了下官方文档也没说如何才能开启日志查询

[Feature]KubeCube如何让第三方用户登录，多租户模型可以和Apache APISIX云原生网关兼容吗？

租户不能删除

创建好的租户支持删除么，在租户管理里面没有找到删除租户的按钮

[Feature] add generic auth method

Is your feature request related to a problem? Please describe.
如果用户已有自己的认证系统，不希望使用KubeCube的认证方法，而希望KubeCube对接自己的认证系统。则需要定义一个通用的第三方认证接口，以便用户可以对接自己的认证系统。

Describe the solution you'd like
定义通用的第三方认证方法：

在启动参数中配置第三方认证地址并开启，则在用户认证（认证中间件进行认证）时，使用第三方认证，而不使用Kubecube自己的认证方法；
要求在第三方认证请求返回时，返回map[string]interface{}格式的body，并且其中需要包含Key=name；并且Kubecube会把返回的response中的header作为整个请求的response header。

Describe alternatives you've considered

Additional context

[Feature] Remove the dependency on modifying k8s apiserver

Why we should remove the dependency on modifying k8s apiserver?

At present, we have to modify the args of k8s apiserver so that auth token can be verified by our auth webhook, but against deployment.

Assumed that there are lots of k8s master nodes, we have to modify each master node one by one. This is unbearable.

A proposal may make sense

Integrate auth proxy in warden.

Warden would parse token for user and do Impersonation with request forwards to k8s apiserver.

[Bug]集群管理添加集群成功，但是查询集群内资源报错

Describe the bug
通过集群管理，成功添加了一个集群，可以查看集群信息。但是查看节点、存储类别、持久存储、网络策略均报错。查看kubecube pod 日志提示请求被管理的集群 apiserver不通

Expected behavior
预期既然添加集群能成功，那么也应该可以正常查看集群节点、存储类别、网络策略

Screenshots

OS: centos 7.4
Browser [e.g. chrome, safari]
Version: kubecube v1.0.0

[Feature]支持带有边缘节点的k8s集群纳入嘛

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
支持带有边缘节点的k8s集群纳入嘛

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[Feature] CubeOptions need validation

Why CubeOptions need validation?
Before running KubeCube, we are supposed to doing options validate, quick exit if validate failed.

	Start = func(c *cli.Context) error {
		if errs := flags.CubeOpts.Validate(); len(errs) > 0 {
			return utilerrors.NewAggregate(errs)
		}

		run(flags.CubeOpts, signals.SetupSignalHandler())

		return nil
	}

How to do it?
We had validate method but in fact doing nothing in each validate func, we need complete it.

// Validate verify options for every component
// todo(weilaaa): complete it
func (s *CubeOptions) Validate() []error {
	var errs []error

	errs = append(errs, s.APIServerOpts.Validate()...)
	errs = append(errs, s.ClientMgrOpts.Validate()...)
	errs = append(errs, s.CtrlMgrOpts.Validate()...)

	return errs
}

func (c *Config) Validate() []error {
	return nil
}

[Bug]前端展示效果建议修复

可以看看下拉组件展示效果，很难去判断选中状态，显示也很不理想，也许是因为只有一条的原因

[Bug]Helm 安装失败

环境：
华为云cce Turbo 集群，版本v1.25。

错误信息截图如下：
01

[Feature]建议新增master节点是否安装kube被调度的相关组件

###生产环境大部分时候

不会安装master被调度的组件，只会是master可以调度其他，所以就是在安装的时候就可以排除不需要的

[Bug][Help] nginx: [emerg] host not found in upstream "audit:8888" in /etc/nginx/nginx.conf:20

Describe the bug
A clear and concise description of what the bug is.
可以使用中文。

使用 All-in-one 文档部署，停在最后一步 deploy kubecube

查看情况发现如下错误

按 v1.1.x 部署
https://www.kubecube.io/docs/installation-guide/all-in-one/

[Bug] member cluster connect failed cause to kubecube startup failure

Describe the bug

member cluster connect failed cause to kubecube startup failure

2021-08-09T10:34:48.179+0800	error	kubernetes/kubernetes.go:81	problem new k8s client: Get "https://10.173.32.130:6443/api?timeout=32s": dial tcp 10.173.32.130:6443: connect: no route to host

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x160106e]

goroutine 1034 [running]:
github.com/kubecube-io/kubecube/pkg/clients/kubernetes.NewClientFor.func1(0xc000df1280, 0x1ea3930, 0xc001c9e1c0)
	/workspace/pkg/clients/kubernetes/kubernetes.go:102 +0x2e
created by github.com/kubecube-io/kubecube/pkg/clients/kubernetes.NewClientFor
	/workspace/pkg/clients/kubernetes/kubernetes.go:101 +0x252

To Reproduce
Steps to reproduce the behavior:

teardown the member cluster
restart kubecube pod

Expected behavior
It's supposed to setting the status of cluster to abnormal until member cluster reconnect kubecube meanwhile refresh InternalCluster

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Bug][Help]已有k8s 1.22.1集群中部署KubeCube不成功?

k8s版本 1.21.1

kube-apiserver.yaml安装官方文档中的kubeapiserver修改 https://www.kubecube.io/docs/installation-guide/install-on-k8s/

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.206.0.10:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - command:
        - kube-apiserver
        - --audit-log-format=json
        - --audit-log-maxage=10
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
        - --audit-log-path=/var/log/audit
        - --audit-policy-file=/etc/cube/audit/audit-policy.yaml
        - --audit-webhook-config-file=/etc/cube/audit/audit-webhook.config
        - --authentication-token-webhook-config-file=/etc/cube/warden/webhook.config
        - --advertise-address=10.206.0.10
        - --allow-privileged=true
        - --authorization-mode=Node,RBAC
        - --client-ca-file=/etc/kubernetes/pki/ca.crt
        - --enable-admission-plugins=NodeRestriction
        - --enable-bootstrap-token-auth=true
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=https://127.0.0.1:2379
        - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
        - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
        - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
        - --requestheader-allowed-names=front-proxy-client
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
        - --requestheader-extra-headers-prefix=X-Remote-Extra-
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-username-headers=X-Remote-User
        - --secure-port=6443
        - --service-account-issuer=https://kubernetes.default.svc.cluster.local
        - --service-account-key-file=/etc/kubernetes/pki/sa.pub
        - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
        - --service-cluster-ip-range=10.16.0.0/12
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
      image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.1
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 8
        httpGet:
          host: 10.206.0.10
          path: /livez
          port: 6443
          scheme: HTTPS
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 15
      name: kube-apiserver
      readinessProbe:
        failureThreshold: 3
        httpGet:
          host: 10.206.0.10
          path: /readyz
          port: 6443
          scheme: HTTPS
        periodSeconds: 1
        timeoutSeconds: 15
      resources:
        requests:
          cpu: 250m
      startupProbe:
        failureThreshold: 24
        httpGet:
          host: 10.206.0.10
          path: /livez
          port: 6443
          scheme: HTTPS
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 15
      volumeMounts:
      - mountPath: /var/log/audit
        name: audit-log
      - mountPath: /etc/cube
        name: cube
        readOnly: true
      - mountPath: /etc/ssl/certs
        name: ca-certs
        readOnly: true
      - mountPath: /etc/ca-certificates
        name: etc-ca-certificates
        readOnly: true
      - mountPath: /etc/kubernetes/pki
        name: k8s-certs
        readOnly: true
      - mountPath: /usr/local/share/ca-certificates
        name: usr-local-share-ca-certificates
        readOnly: true
      - mountPath: /usr/share/ca-certificates
        name: usr-share-ca-certificates
        readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /var/log/audit
      type: DirectoryOrCreate
    name: audit-log
  - hostPath:
      path: /etc/cube
      type: DirectoryOrCreate
    name: cube
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

【错误报告】

[Bug]新版的ingress不能直接使用

直接通过界面配置，会报错
找不到ingressClassName:
由于公司是云服务器，使用了两台有外网的做实验
123.123.123.111 10.10.10.31 （添加了虚拟网卡将外网绑定到主机）
123.123.123.222 10.10.10.32 （添加了虚拟网卡将外网绑定到主机）
10.10.10.31 直接使用的all-in-one模式安装
10.10.10.32 node-join-master
最总结果kube get node显示结果为
10.10.10.31 master
123.123.123.222 node （这里估计填写node ip 10.10.10.32 可以使用内网ip也应该没什么问题）

上面括弧重的猜测已经测试：
KUBERNETES_BIND_ADDRESS="10.10.10.32" node-join-master 的时候显示的也是外网Ip 123.123.123.222
我的网卡信息为

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:a0:9e:2b brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.32/24 brd 10.10.10.255 scope global dynamic eth0
       valid_lft 25917702sec preferred_lft 25917702sec
    inet 123.123.123.222/24 brd 123.123.123.255 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea0:9e2b/64 scope link 
       valid_lft forever preferred_lft forever
 eth0:1 为追加虚拟网卡，绑定我的外网ip

然后添加 deployment dep-ng nginx->80
添加 service svc-ng dep-ng 80->80
添加 ingress ing-ng svc-ng 80 域名a.cn 转发规则“/”
将域名 a.cn解析到123.123.123.111
发现不能访问
查看ingress日志说找不到ingressClassName
修改ingress ing-ng的yml配置，添加了ingressClassName:nginx
再查看日志，发现没有错误日志，但是域名依然无法访问

1.7的安装文档弄下，到了1.4怎么就没了

1.7的安装文档弄下啊，到了1.4怎么就没了

[Bug]节点添加脚本错误相关问题

问题一：向集群添加工作节点时执行脚本install.sh报错
2021-08-18 15:45:41 INFO get docker binary from local /bin/mv: cannot stat ‘/etc/kubecube/packages/docker-ce/linux/static/stable/x86_64/docker-19.03.8.tgz’: No such file or directory 2021-08-18 15:45:41 ERROR install kubernetes failed

真正packages目录是在packages-master中：
[root@test-ec2 x86_64]# pwd /etc/kubecube/packages-master/docker-ce/linux/static/stable/x86_64 [root@gtlm-ec2 x86_64]# ls docker-19.03.8.tgz

问题二：添加新节点时，给的步骤链接404
链接：https://www.kubecube.io/docs/部署指南/添加节点/#向集群添加工作节点

问题三：
创建新集群时，过程与文档：https://www.kubecube.io/docs/installation-guide/add-member-k8s 完全不符！
这个问题新手遇到容易发狂！

[Feature] add debug script and makefile

I'd like to deploy and debug kubecube in local. I want an easier way to make deploy environment and debug. Something like install script and makefile is needed. manifests needed below:

dependence manifests(crds, svc, rbac and so on) install and uninstall script
local running makefile
image build makefile

[Feature] 支持网络策略的配置或者支持不同租户下的网络隔离