Comments (11)
snpyeso
commented on May 27, 2024
8
@ethfoo @zhujf1989 @tiancandevloper @hzliangbin 可以看一看么
from kubecube.
snpyeso
commented on May 27, 2024
5
重新安装kubecube，使用的最新版，全部开启common里的插件和pivot-cluster里的插件，进入日志查询,依然 request elasticsearch fail
安装结果也和上面差不多
from kubecube.
tiancandevloper
commented on May 27, 2024
https://www.kubecube.io/docs/user-guide/administration/audit/
可以参考这篇文章，将logseer和logagent开启
from kubecube.
zhujf1989
commented on May 27, 2024
可以参考文档：https://www.kubecube.io/docs/installation-guide/enable-plugins/
修改HotPlug的配置, 将logseer及logagent enable
- name: logseer ## 日志管理组件
namespace: logseer
pkgName: logseer-v1.0.0.tgz
status: enabled ## 启停标识：这里disabled为禁用
- name: logagent ## 日志采集代理组件
namespace: logagent
pkgName: logagent-v1.0.0.tgz
status: enabled ## 启停标识：这里enabled为启用
env: | ## 环境变量
clustername: "{{.cluster}}" ## {{.cluster}} 程序会自动注入集群名字替换
from kubecube.
snpyeso
commented on May 27, 2024
可以参考文档：https://www.kubecube.io/docs/installation-guide/enable-plugins/ 修改HotPlug的配置, 将logseer及logagent enable
- name: logseer ## 日志管理组件 namespace: logseer pkgName: logseer-v1.0.0.tgz status: enabled ## 启停标识：这里disabled为禁用 - name: logagent ## 日志采集代理组件 namespace: logagent pkgName: logagent-v1.0.0.tgz status: enabled ## 启停标识：这里enabled为启用 env: | ## 环境变量 clustername: "{{.cluster}}" ## {{.cluster}} 程序会自动注入集群名字替换
这篇文章我看过，但是我的hotplug状态为fail 不知道有没有影响，文档中是runing,看了一会儿也没发现如何让它变成running状态
[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME PHASE AGE
common fail 2d2h
pivot-cluster fail 2d2h将logseer及logagent enable 之后
kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
cqdxdevspace dep-prepare-6596d79cf-mcmqw 1/1 Running 0 19h
elasticsearch elasticsearch-master-0 1/1 Running 0 2d
hnc-system hnc-controller-manager-6d47799b76-kp4wl 1/1 Running 1 (2d ago) 2d
ingress-nginx ingress-nginx-admission-create-ggrhg 0/1 Completed 0 2d
ingress-nginx ingress-nginx-admission-patch-chxhk 0/1 Completed 1 2d
ingress-nginx ingress-nginx-controller-776d88d5c9-mthdn 1/1 Running 0 2d
kube-system calico-kube-controllers-d84f7b88b-krrjs 1/1 Running 0 2d
kube-system calico-node-b55ds 1/1 Running 0 2d
kube-system coredns-65c54cc984-l4tpt 1/1 Running 0 2d
kube-system coredns-65c54cc984-rq6b8 1/1 Running 0 2d
kube-system etcd-localhost.localdomain 1/1 Running 8 2d
kube-system kube-apiserver-localhost.localdomain 1/1 Running 0 26h
kube-system kube-controller-manager-localhost.localdomain 1/1 Running 28 (26h ago) 2d
kube-system kube-proxy-7bgj5 1/1 Running 0 2d
kube-system kube-scheduler-localhost.localdomain 1/1 Running 22 (26h ago) 2d
kube-system metrics-server-6d69ff96c6-7s27b 1/1 Running 3 (26h ago) 2d
kubecube-monitoring kubecube-monitoring-kube-state-metrics-6cf5c8c6d4-dqv2t 1/1 Running 0 2d
kubecube-monitoring kubecube-monitoring-operator-5f84949849-sjtwq 1/1 Running 0 2d
kubecube-monitoring kubecube-monitoring-prometheus-node-exporter-c9cdt 1/1 Running 0 2d
kubecube-monitoring kubecube-thanos-query-86f669566f-ndwm7 1/1 Running 0 2d
kubecube-monitoring kubecube-thanos-receive-0 1/1 Running 0 2d
kubecube-monitoring prometheus-kubecube-monitoring-prometheus-0 2/2 Running 0 2d
kubecube-system audit-78b7d4cf7d-4h2hq 1/1 Running 0 2d
kubecube-system cloud-shell-5c87948759-4zcwq 1/1 Running 0 2d
kubecube-system frontend-7cbccf84fb-vh5hv 1/1 Running 0 2d
kubecube-system kubecube-78b59d768d-tpssk 1/1 Running 3 (26h ago) 2d
kubecube-system warden-d876d9fd8-969q5 1/1 Running 0 2d
kubecube-system webconsole-7d7f989996-wvh2r 1/1 Running 3 (26h ago) 2d
local-path-storage local-path-provisioner-86b7fd94b5-gh2wl 1/1 Running 3 (26h ago) 2d
logagent filebeat-dhwr5 2/2 Running 0 6m16s
没有发现 logseer相关的pod
查看common yaml 发现logseer 为uninstalled
spec:
component:
-
name: audit
status: disabled
-
name: logseer
namespace: logseer
pkgName: logseer-v1.0.0.tgz
status: enabled
-
env: "clustername: \"{{.cluster}}\"\n"
name: logagent
namespace: logagent
pkgName: logagent-v1.0.0.tgz
status: enabled
-
name: elasticsearch
namespace: elasticsearch
pkgName: elasticsearch-7.8.1.tgz
status: enabled
-
env: "grafana:\n enabled: false\nprometheus:\n prometheusSpec:\n externalLabels:\n cluster: \"{{.cluster}}\"\n remoteWrite:\n - url: http://10.10.10.44:31291/api/v1/receive\n"
name: kubecube-monitoring
namespace: kubecube-monitoring
pkgName: kubecube-monitoring-15.4.12.tgz
status: enabled
-
name: kubecube-thanos
namespace: kubecube-monitoring
pkgName: thanos-3.18.0.tgz
status: disabled
status:
phase: fail
results:
-
message: 'audit is disabled'
name: audit
result: success
status: disabled
-
message: uninstalled
name: logseer
result: success
status: disabled
-
message: 'release is running'
name: logagent
result: success
status: enabled
-
message: 'release is running'
name: elasticsearch
result: success
status: enabled
-
message: 'helm install fail, cannot re-use a name that is still in use'
name: kubecube-monitoring
result: fail
status: enabled
-
message: 'release is running'
name: kubecube-thanos
result: success
status: enabled检查下发现 有common和pivot-cluster 的热插拔，发现common 和 pivot-cluster都有 logseer配置，小白都先开启吧，然后再开启elasticsearch,
呃…… 这次错误提示还是蛮明显的，但是不知道如何修复
{"reason":400,"code":400,"message":"request elasticsearch fail"}
检查下elasticsearch的服务 发现一切正常
[root@localhost ~]# kubectl get all -n elasticsearch
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 147m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.108.228.229 <none> 9200/TCP,9300/TCP 147m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 147m
NAME READY AGE
statefulset.apps/elasticsearch-master 1/1 147mfrom kubecube.
zhujf1989
commented on May 27, 2024
/cc @weilaaa
from kubecube.
snpyeso
commented on May 27, 2024
@fusidamj @hzliangbin @tiancandevloper @zhujf1989
from kubecube.
ethfoo
commented on May 27, 2024
麻烦发下请求的时候logseer的日志
from kubecube.
snpyeso
commented on May 27, 2024
麻烦发下请求的时候logseer的日志
好的稍等，我得重装一下，因为我尝试自己修改已经有点改得记不到哪里要恢复了
from kubecube.
snpyeso
commented on May 27, 2024
@ethfoo
安装了两台机器
[root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME STATUS ROLES AGE VERSION
zpfrltgup4tujpi1-0001.novalocal Ready control-plane,master 7m3s v1.23.5
zpfrltgup4tujpi1.novalocal Ready node 32s v1.23.5进入空间配置好基本环境 新增租户空间等
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME STATUS AGE
default Active 13m
elasticsearch Active 12m
hnc-system Active 12m
ingress-nginx Active 12m
kube-node-lease Active 13m
kube-public Active 13m
kube-system Active 13m
kubecube-monitoring Active 13m
kubecube-project-wordpress Active 113s
kubecube-system Active 12m
kubecube-tenant-tenant Active 2m2s
local-path-storage Active 12m
wordpresspace Active 50s添加一个nginx 测试deployment
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n wordpresspace
NAME READY STATUS RESTARTS AGE
pod/dep-ng-5f6b6f4778-j5vb2 1/1 Running 0 62s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dep-ng 1/1 1 1 62s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dep-ng-5f6b6f4778 1 1 1 62s开启全部热插件
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name: common
Namespace:
Labels: <none>
Annotations: kubecube.io/sync: true
API Version: hotplug.kubecube.io/v1
Kind: Hotplug
Metadata:
Creation Timestamp: 2022-07-13T01:57:47Z
Generation: 2
Managed Fields:
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kubecube.io/sync:
f:spec:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-13T01:57:47Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:phase:
f:results:
Manager: warden
Operation: Update
Subresource: status
Time: 2022-07-13T02:04:13Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:component:
Manager: Mozilla
Operation: Update
Time: 2022-07-13T02:14:01Z
Resource Version: 6058
UID: fc94d067-3712-4907-aca6-48d7f4095314
Spec:
Component:
Name: audit
Status: enabled
Name: logseer
Namespace: logseer
Pkg Name: logseer-v1.0.0.tgz
Status: enabled
Env: clustername: "{{.cluster}}"
Name: logagent
Namespace: logagent
Pkg Name: logagent-v1.0.0.tgz
Status: enabled
Name: elasticsearch
Namespace: elasticsearch
Pkg Name: elasticsearch-7.8.1.tgz
Status: enabled
Env: grafana:
enabled: false
prometheus:
prometheusSpec:
externalLabels:
cluster: "{{.cluster}}"
remoteWrite:
- url: http://172.31.0.171:31291/api/v1/receive
Name: kubecube-monitoring
Namespace: kubecube-monitoring
Pkg Name: kubecube-monitoring-15.4.12.tgz
Status: enabled
Name: kubecube-thanos
Namespace: kubecube-monitoring
Pkg Name: thanos-3.18.0.tgz
Status: enabled
Status:
Phase: fail
Results:
Message: audit is enabled
Name: audit
Result: success
Status: enabled
Message: release is running
Name: logseer
Result: success
Status: enabled
Message: release is running
Name: logagent
Result: success
Status: enabled
Message: release is running
Name: elasticsearch
Result: success
Status: enabled
Message: helm install fail, cannot re-use a name that is still in use
Name: kubecube-monitoring
Result: fail
Status: enabled
Message: release is running
Name: kubecube-thanos
Result: success
Status: enabled
Events: <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name: pivot-cluster
Namespace:
Labels: <none>
Annotations: kubecube.io/sync: true
API Version: hotplug.kubecube.io/v1
Kind: Hotplug
Metadata:
Creation Timestamp: 2022-07-13T01:57:47Z
Generation: 2
Managed Fields:
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kubecube.io/sync:
f:spec:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-13T01:57:47Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:phase:
f:results:
Manager: warden
Operation: Update
Subresource: status
Time: 2022-07-13T02:04:15Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:component:
Manager: Mozilla
Operation: Update
Time: 2022-07-13T02:14:09Z
Resource Version: 6066
UID: 098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
Component:
Name: logseer
Status: enabled
Env: grafana:
enabled: true
prometheus:
prometheusSpec:
externalLabels:
cluster: "{{.cluster}}"
remoteWrite:
- url: http://kubecube-thanos-receive:19291/api/v1/receive
Name: kubecube-monitoring
Env: receive:
tsdbRetention: 7d
replicaCount: 1
replicationFactor: 1
Name: kubecube-thanos
Status: enabled
Status:
Phase: fail
Results:
Message: audit is enabled
Name: audit
Result: success
Status: enabled
Message: release is running
Name: logseer
Result: success
Status: enabled
Message: release is running
Name: logagent
Result: success
Status: enabled
Message: release is running
Name: elasticsearch
Result: success
Status: enabled
Message: helm install fail, cannot re-use a name that is still in use
Name: kubecube-monitoring
Result: fail
Status: enabled
Message: release is running
Name: kubecube-thanos
Result: success
Status: enabled
配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng
点击日志查询
查看logseer日志
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer
NAME READY STATUS RESTARTS AGE
pod/logseer-657645d857-w5rv5 1/1 Running 0 6m11s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/logseer ClusterIP 10.105.118.70 <none> 80/TCP 6m11s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/logseer 1/1 1 1 6m11s
NAME DESIRED CURRENT READY AGE
replicaset.apps/logseer-657645d857 1 1 1 6m11s
##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:52 INFO - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
"size": 50,
"from": 0,
"query": {
"bool" : {
"filter" : [
{"term": {"cluster_name" : "pivot-cluster"}},
{"term": {"namespace" : "wordpresspace"}}
],
"must" : [
{
"query_string" : {
"default_field" : "message",
"query" : "*"
}
},
{
"range" : {
"@timestamp" : {
"gte" : 1657675115512,
"lte" : 1657678715512,
"format": "epoch_millis"
}
}
}
]
}
},
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "1d",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
}
}
},
"highlight" : {
"fields" : {
"message" : {}
},
"fragment_size": 2147483647
},
"sort" : [
{ "@timestamp" : "asc"}
],
"_source" : {
"excludes": "tags"
},
"timeout": "30000ms"
}
2022-07-13 10:22:12.384 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:65 ERROR - request elasticsearch exception: {}
java.net.ConnectException: null
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
from kubecube.
snpyeso
commented on May 27, 2024
@ethfoo 安装了两台机器
[root@zpfrltgup4tujpi1 manifests]# kubectl get node NAME STATUS ROLES AGE VERSION zpfrltgup4tujpi1-0001.novalocal Ready control-plane,master 7m3s v1.23.5 zpfrltgup4tujpi1.novalocal Ready node 32s v1.23.5进入空间配置好基本环境 新增租户空间等
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns NAME STATUS AGE default Active 13m elasticsearch Active 12m hnc-system Active 12m ingress-nginx Active 12m kube-node-lease Active 13m kube-public Active 13m kube-system Active 13m kubecube-monitoring Active 13m kubecube-project-wordpress Active 113s kubecube-system Active 12m kubecube-tenant-tenant Active 2m2s local-path-storage Active 12m wordpresspace Active 50s添加一个nginx 测试deployment
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n wordpresspace NAME READY STATUS RESTARTS AGE pod/dep-ng-5f6b6f4778-j5vb2 1/1 Running 0 62s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/dep-ng 1/1 1 1 62s NAME DESIRED CURRENT READY AGE replicaset.apps/dep-ng-5f6b6f4778 1 1 1 62s开启全部热插件
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common Name: common Namespace: Labels: <none> Annotations: kubecube.io/sync: true API Version: hotplug.kubecube.io/v1 Kind: Hotplug Metadata: Creation Timestamp: 2022-07-13T01:57:47Z Generation: 2 Managed Fields: API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:kubecube.io/sync: f:spec: Manager: kubectl-client-side-apply Operation: Update Time: 2022-07-13T01:57:47Z API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:phase: f:results: Manager: warden Operation: Update Subresource: status Time: 2022-07-13T02:04:13Z API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:component: Manager: Mozilla Operation: Update Time: 2022-07-13T02:14:01Z Resource Version: 6058 UID: fc94d067-3712-4907-aca6-48d7f4095314 Spec: Component: Name: audit Status: enabled Name: logseer Namespace: logseer Pkg Name: logseer-v1.0.0.tgz Status: enabled Env: clustername: "{{.cluster}}" Name: logagent Namespace: logagent Pkg Name: logagent-v1.0.0.tgz Status: enabled Name: elasticsearch Namespace: elasticsearch Pkg Name: elasticsearch-7.8.1.tgz Status: enabled Env: grafana: enabled: false prometheus: prometheusSpec: externalLabels: cluster: "{{.cluster}}" remoteWrite: - url: http://172.31.0.171:31291/api/v1/receive Name: kubecube-monitoring Namespace: kubecube-monitoring Pkg Name: kubecube-monitoring-15.4.12.tgz Status: enabled Name: kubecube-thanos Namespace: kubecube-monitoring Pkg Name: thanos-3.18.0.tgz Status: enabled Status: Phase: fail Results: Message: audit is enabled Name: audit Result: success Status: enabled Message: release is running Name: logseer Result: success Status: enabled Message: release is running Name: logagent Result: success Status: enabled Message: release is running Name: elasticsearch Result: success Status: enabled Message: helm install fail, cannot re-use a name that is still in use Name: kubecube-monitoring Result: fail Status: enabled Message: release is running Name: kubecube-thanos Result: success Status: enabled Events: <none> [root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster Name: pivot-cluster Namespace: Labels: <none> Annotations: kubecube.io/sync: true API Version: hotplug.kubecube.io/v1 Kind: Hotplug Metadata: Creation Timestamp: 2022-07-13T01:57:47Z Generation: 2 Managed Fields: API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:kubecube.io/sync: f:spec: Manager: kubectl-client-side-apply Operation: Update Time: 2022-07-13T01:57:47Z API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:phase: f:results: Manager: warden Operation: Update Subresource: status Time: 2022-07-13T02:04:15Z API Version: hotplug.kubecube.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:component: Manager: Mozilla Operation: Update Time: 2022-07-13T02:14:09Z Resource Version: 6066 UID: 098fe080-bbc9-40db-a7c6-b8e36290191e Spec: Component: Name: logseer Status: enabled Env: grafana: enabled: true prometheus: prometheusSpec: externalLabels: cluster: "{{.cluster}}" remoteWrite: - url: http://kubecube-thanos-receive:19291/api/v1/receive Name: kubecube-monitoring Env: receive: tsdbRetention: 7d replicaCount: 1 replicationFactor: 1 Name: kubecube-thanos Status: enabled Status: Phase: fail Results: Message: audit is enabled Name: audit Result: success Status: enabled Message: release is running Name: logseer Result: success Status: enabled Message: release is running Name: logagent Result: success Status: enabled Message: release is running Name: elasticsearch Result: success Status: enabled Message: helm install fail, cannot re-use a name that is still in use Name: kubecube-monitoring Result: fail Status: enabled Message: release is running Name: kubecube-thanos Result: success Status: enabled配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng
点击日志查询
查看logseer日志
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer NAME READY STATUS RESTARTS AGE pod/logseer-657645d857-w5rv5 1/1 Running 0 6m11s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/logseer ClusterIP 10.105.118.70 <none> 80/TCP 6m11s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/logseer 1/1 1 1 6m11s NAME DESIRED CURRENT READY AGE replicaset.apps/logseer-657645d857 1 1 1 6m11s ##日志如下 2022-07-13 10:22:11.377 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:52 INFO - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: { "size": 50, "from": 0, "query": { "bool" : { "filter" : [ {"term": {"cluster_name" : "pivot-cluster"}}, {"term": {"namespace" : "wordpresspace"}} ], "must" : [ { "query_string" : { "default_field" : "message", "query" : "*" } }, { "range" : { "@timestamp" : { "gte" : 1657675115512, "lte" : 1657678715512, "format": "epoch_millis" } } } ] } }, "aggs": { "2": { "date_histogram": { "field": "@timestamp", "interval": "1d", "time_zone": "Asia/Shanghai", "min_doc_count": 1 } } }, "highlight" : { "fields" : { "message" : {} }, "fragment_size": 2147483647 }, "sort" : [ { "@timestamp" : "asc"} ], "_source" : { "excludes": "tags" }, "timeout": "30000ms" } 2022-07-13 10:22:12.384 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:65 ERROR - request elasticsearch exception: {} java.net.ConnectException: null at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53) at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154) at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79) at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: null at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
期待回复
from kubecube.
Related Issues (20)
- [Bug] I installed all in one,bug there's some problerm with me HOT 2
- [Bug]新版的ingress不能直接使用 HOT 10
- 租户不能删除 HOT 1
- [Bug]日志输出乱码 HOT 2
- [Bug]按照文档一路安装，从1.2-到1.4【logseer】 真用不起 HOT 2
- [Bug] 部署脚本缺少必要的终止判断 HOT 1
- 1.7的安装文档弄下，到了1.4怎么就没了 HOT 4
- kubecube all in one 安装方式的一些疑问 HOT 1
- [Feature]hnc support hncresourcequota，是不是可以使用最新的来更新了？ HOT 1
- Helm安装kubecube问题 HOT 2
- 在已有k8s集群中安装kubecube后，可以在kubecube中管理集群中已有的资源吗？ HOT 1
- [Bug] helm安装时 Nodeport 和已经存在的Ingress的port冲突且无法规避 HOT 2
- 创建项目时报错 HOT 1
- [Bug]使用通用认证方式接入第三方认证时用户在kubecube中不会自动创建 HOT 1
- [Bug]Helm 安装失败 HOT 2
- [Feature] 组件监控里面为什么没有信息 HOT 1
- [Feature]官网提供的安装教程以及其他信息很久没更新 HOT 3
- [Feature]支持带有边缘节点的k8s集群纳入嘛
- [Bug]跟ceph存储集群混合部署会无法正常运行kubecube
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubecube.