Giter Site home page Giter Site logo

kube-eventer's Introduction

kube-eventer

kube-eventer emit kubernetes events to sinks

Overview

License Build Status Codecov

kube-eventer is an event emitter that sends kubernetes events to sinks(.e.g, dingtalk,sls,kafka and so on). The core design concept of kubernetes is state machine. So there will be Normal events when transfer to desired state and Warning events occur when to unexpected state. kube-eventer can help to diagnose, analysis and alarm problems.

Architecture diagram

Architecture diagram of kube-eventer

Usage

Here is a demo of kube-eventer with dingtalk sink. Dingtalk is an All-in-one Mobile Workplace like slack. Dingtalk sink can get kubernetes events from eventer. You can do chatops with it. Try it following the steps below.

  1. Get dingtalk token Add a chatbot in a dingtalk group and get bot token.

  1. Install eventer and configure sink
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: kube-eventer
  name: kube-eventer
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eventer
  template:
    metadata:
      labels:
        app: kube-eventer
      annotations:	
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: kube-eventer
      containers:
        - image: registry.aliyuncs.com/acs/kube-eventer:v1.2.7-ca03be0-aliyun
          name: kube-eventer
          command:
            - "/kube-eventer"
            - "--source=kubernetes:https://kubernetes.default"
            ## .e.g,dingtalk sink demo
            - --sink=dingtalk:[your_webhook_url]&label=[your_cluster_id]&level=[Normal or Warning(default)]
          env:
          # If TZ is assigned, set the TZ value as the time zone
          - name: TZ
            value: "Asia/Shanghai" 
          volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
            - name: zoneinfo
              mountPath: /usr/share/zoneinfo
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
            limits:
              cpu: 500m
              memory: 250Mi
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: zoneinfo
          hostPath:
            path: /usr/share/zoneinfo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-eventer
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - events
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-eventer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eventer
subjects:
  - kind: ServiceAccount
    name: kube-eventer
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-eventer
  namespace: kube-system
  1. View events in dingtalk

Sink Configure

Supported Sinks:

Sink Name Description
dingtalk sink to dingtalk bot
sls sink to alibaba cloud sls service
elasticsearch sink to elasticsearch
honeycomb sink to honeycomb
influxdb sink to influxdb
kafka sink to kafka
mysql sink to mysql database
wechat sink to wechat
webhook sink to webhook
mongodb sink to mongodb

Contributing

Please check CONTRIBUTING.md

You can also ask for help in our Dingtalk group. dingtalk group1 (full): 30612594 dingtalk group2: 5605020816

License

This software is released under the Apache 2.0 license.

kube-eventer's People

Contributors

chivalryq avatar daixijun avatar dependabot[bot] avatar east4ming avatar imdingtalk avatar jarhmj avatar jemygraw avatar jiazhiguang avatar keyofspectator avatar king5211 avatar lazywhite avatar liuming-dev avatar lwabish avatar michaelcheungdk avatar muchcontact avatar oliverwoo avatar onecricketeer avatar ringtail avatar rralcala avatar shabicheng avatar stephenliuya avatar stormgbs avatar wangjc0216 avatar wind2008hxy avatar zalmanzhao avatar zeusro avatar zhouxinyu avatar zjj2wry avatar zp-alwayswin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kube-eventer's Issues

ES Parameter problem

When using es as the event receiver, When you specify the sink parameter, if you do not set use_namespace,The following error will be displayed:
W1010 03:22:00.006045 1 driver.go:94] Failed to export data to ElasticSearch sink: elastic: Error 400 (Bad Request): Invalid index name [heapster-2019-10-10 07:21:55 +0000 UTC], must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?] [type=invalid_index_name_exception]
So,if this parameter must be specified, please help to join the es help file. @rralcala @ringtail

multi sink, a sink namespaces filter not work

C1BE1DDF-8D7E-4eaf-8BD7-77179FE6BEE3

1.两个webhook都有推送信息,没问题
2.但是其中一个sink配置了只推两个命名空间,现象却是推全量,全部命名空间的事件都推了

a sink namespaces filter not work

release v1.1.0

  • Add dingtalk markdown support
  • Add timezone support
  • Fix kafka sink bug
  • Fix dingtalk bug #18

逻辑上的误报 和 告警频繁问题,求助。

试用了一段时间, 发现 在更新 Deployment 的时候 Pod伸缩的时候 , 也会 报警, 只要 Pod的状态 改变都会触发 报警, 能不能 判断一下, 区分下 是意料之中的状态改变, 还是 应该 报警的状态。

是否能延迟报警, 将 状态改变的Pod 放入待观察的队列里, 进行进一步的赛选呢。

太过频繁的报警 让我们 无法区分 到底 要不要去处理

使用您提供的yaml示例修改后无法部署到k8s中

apiVersion: apps/v1beta2
kind: Deployment
metadata:
labels:
name: kube-eventer
name: kube-eventer
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: kube-eventer
template:
metadata:
labels:
app: kube-eventer
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
dnsPolicy: ClusterFirstWithHostNet
serviceAccount: kube-eventer
containers:
- image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.1.0-63e7f98-aliyun
name: kube-eventer
command:
- "/kube-eventer"
- "--source=kubernetes:https://kubernetes.default"
## .e.g,dingtalk sink demo
- --sink=dingtalk:[your_webhook_url]&label=[your_cluster_id]&level=[Normal or Warning(default)]
env:
# If TZ is assigned, set the TZ value as the time zone
- name: TZ
value: America/New_York
volumeMounts:
- name: localtime
mountPath: /etc/localtime
readOnly: true
- name: zoneinfo
mountPath: /usr/share/zoneinfo
readOnly: true
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 500m
memory: 250Mi
volumes:
- name: localtime
hostPath:
path: /etc/localtime
- name: zoneinfo
hostPath:
path: /usr/share/zoneinfo

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-eventer
rules:

  • apiGroups:
    • ""
      resources:
    • events
      verbs:
    • get
    • list
    • watch

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
name: kube-eventer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-eventer
subjects:

  • kind: ServiceAccount
    name: kube-eventer
    namespace: kube-system

apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-eventer
namespace: kube-system

提示报错
image
当前kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

kubeletes-dashboard版本是:
image

钉钉推送失败

配置信息

image: registry.cn-beijing.aliyuncs.com/acs/kube-eventer-amd64:v1.0.0-d9898e1-aliyun
name: kube-eventer
command
  - "/kube-eventer"
  - "--source=kubernetes:https://kubernetes.default"
  - "--sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=<My-Token>&label=Kubernetes-Test&level=Normal"

日志信息

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xad615f]

goroutine 28 [running]:
github.com/AliyunContainerService/kube-eventer/sinks/dingtalk.(*DingTalkSink).Ding(0xc00030c380, 0xc000144780)
	/src/github.com/AliyunContainerService/kube-eventer/sinks/dingtalk/dingtalk.go:151 +0x30f
github.com/AliyunContainerService/kube-eventer/sinks/dingtalk.(*DingTalkSink).ExportEvents(0xc00030c380, 0xc000102720)
	/src/github.com/AliyunContainerService/kube-eventer/sinks/dingtalk/dingtalk.go:93 +0x97
github.com/AliyunContainerService/kube-eventer/sinks.export(0x17ea640, 0xc00030c380, 0xc000102720)
	/src/github.com/AliyunContainerService/kube-eventer/sinks/manager.go:145 +0x91
github.com/AliyunContainerService/kube-eventer/sinks.NewEventSinkManager.func1(0x17ea640, 0xc00030c380, 0xc00012f320, 0xc00012f380)
	/src/github.com/AliyunContainerService/kube-eventer/sinks/manager.go:77 +0x216
created by github.com/AliyunContainerService/kube-eventer/sinks.NewEventSinkManager
	/src/github.com/AliyunContainerService/kube-eventer/sinks/manager.go:73 +0x199

我们想把eventer打到elasticsearch,但是出现了问题

目前我们使用的是elasticsearch6,配完以后,日志发送的时候报错说不支持string。

W0813 03:37:00.073560 1 driver.go:90] Failed to export data to ElasticSearch sink: elastic: Error 400 (Bad Request): Failed to parse mapping [general]: No handler for type [string] declared on field [cluster_name] [type=mapper_parsing_exception]

输出到ES报错

W0904 05:46:30.011489 1 driver.go:90] Failed to export data to ElasticSearch sink: elastic: Error 400 (Bad Request): Failed to parse mapping [general]: No handler for type [string] declared on field [cluster_name] [type=mapper_parsing_exception]

写入到kafka ,pod无法启动,topic已经提前建好

Failed to create {"kafka { false brokers=10.204.57.32:9092&brokers=10.204.57.33:9092&brokers=10.204.57.34:9092&eventstopic=eventstopic" }} sink: Sink not recognized: "kafka
F0720 13:02:41.973843 1 eventer.go:90] No available sink to use

yaml文件。

command:
- /kube-eventer
- "--source=kubernetes:https://kubernetes.default"
- --sink="kafka:?brokers=10.204.57.32:9092&brokers=10.204.57.33:9092&brokers=10.204.57.34:9092&eventstopic=eventstopic"

写错了吗/

建议增加报警频率设置

您好,建议增加报警的频率设置,不然集群内的pod出现问题,不论设置的Warining还是Normal,钉钉上都会连续收到报警,而客户希望一种情况只收到一条报警就好了

部署kube-eventer后,etcd报错增多

您好,经过多次测试,只要部署kube-eventer以后,etcd的报错就增多,关闭后,就正常。请问这个是怎么回事?
image

Prometheus查询语法:

sum by(grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job="etcd"}[5m])) / sum by(grpc_service, grpc_method) (rate(grpc_server_handled_total{job="etcd"}[5m])) > 0.01

[Proposal] Aggregate events with same Ref and Reson

Due to #66 #42 . We decide to support aggregate events with same ref and reason. This is especially when you have a very large deployment can not be scheduled because of some reasons such as lack of resource.

deployment A - >
       Pod A 
       Pod B 
       Pod C 
       ... 
       Pod Z

There would be so many events emitted.

Name: Pod A Reason: Pending Ref: Deploy A Message: inefficient cpu
Name: Pod B Reason: Pending Ref: Deploy A Message: inefficient cpu
Name: Pod C Reason: Pending Ref: Deploy A Message: inefficient cpu
...
Name: Pod Z Reason: Pending Ref: Deploy A Message: inefficient cpu

This proposal would aggregate event in on batch like below

Name: Pod A and other 25 Pods Reason: Pending Ref: Deploy A Message: inefficient cpu

我想添加一个markdown富文本格式的钉钉推送

现状

目前钉钉推送的格式类似是这样的

<clusterid>
Level:Warning 
Namespace:default 
Name:pod-abcd
Message:Port 31666 was assigned to multiple services; please recreate service 
Reason:PortAlreadyAllocated 
Timestamp:3019-07-01 08:38:03 +0000 UTC

我想实现的效果:

Level: Warning

Namespace: default

Name: pod-abcd

Message: Port 31666 was assigned to multiple services; please recreate service

Reason:PortAlreadyAllocated

Timestamp:3019-07-01 08:38:03 +0000 UTC

根据你们控制台的URL规律以及event的事件类型,显示对应的URL

Deployment/Pod

https://cs.console.aliyun.com/#/k8s/deployment/detail///default/deployment1/pods

StatefulSet

https://cs.console.aliyun.com/#/k8s/statefulset/detail///default/statefulset1/pods

DaemonSet

https://cs.console.aliyun.com/#/k8s/daemonset/detail///kube-system/daemonset1/pods

todo

  1. 需要把label预设为集群ID,或者追加一个clusterID参数
  2. 需要在容器的启动命令追加参数,传入region和msgtype

参考链接

添加钉钉报警这个目前只支持阿里云的k8s吗

--sink=dingtalk:[your_webhook_url]&label=[your_cluster_id]&level=[可选参数:Normal或者Warning,默认值为:Warning]
这个代码中的your_cluster_id 只能是阿里云的k8s吗,如果是自己搭建的k8s这个字段如何写

镜像无法拉取

docker pull registry.aliyuncs.com/acs/kube-eventer-amd64:v1.1.0-c93a835-aliyun

提示:
Error response from daemon: Get https://registry.aliyuncs.com/v2/acs/kube-eventer-amd64/manifests/latest: Get https://dockerauth.cn-hangzhou.aliyuncs.com/auth?scope=repository%3Aacs%2Fkube-eventer-amd64%3Apull&service=registry.aliyuncs.com%3Acn-hangzhou%3A26842: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

麻烦重新提供一下镜像地址,谢谢。

README未更新

readme上面的deployment command用的还是eventer, 并没有更改为kube-eventer, 镜像版本也比较低, 建议和deploy/ 目录保持一致

支持私有化环境的k8s集群么

配置文件中的 label=[your_cluster_id] 这个 your_cluster_id的值 是阿里云的环境中才有的么。可以使用该项目用于自己私有化环境搭建的k8s集群么

Mysql sink参数问题

在指定mysql sink的参数时,应指定类似下方的格式:
--sink=mysql:?root:123456@tcp(172.22.2.192:3306)/kube_event?charset=utf8
而且,应该说明,用户应该提前自建一个kube_event的数据库,并在库中自建kube_event的数据表,不然会报错。

no kind "Deployment" is registered for version "apps/v1beta2

no kind "Deployment" is registered for version "apps/v1beta2"                        
no kind "ClusterRole" is registered for version "rbac.authorization.k8s.io/v1"       
no kind "ClusterRoleBinding" is registered for version "rbac.authorization.k8s.io/v1"

根据项目readme部署报错 集群版本 1.12.6-aliyun.1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.