apache/incubator-eventmesh

EventMesh is a new generation serverless event middleware for building distributed event-driven applications.

📦 Documentation | 📔 Examples | ⚙️ Roadmap | 🌐 简体中文

Apache EventMesh

Apache EventMesh is a new generation serverless event middleware for building distributed event-driven applications.

EventMesh Architecture

EventMesh Dashboard

Features

Apache EventMesh has a vast amount of features to help users achieve their goals. Let us share with you some of the key features EventMesh has to offer:

• Built around the CloudEvents specification.
• Rapidty extendsible interconnector layer connectors using openConnect such as the source or sink of Saas, CloudService, and Database etc.
• Rapidty extendsible storage layer such as Apache RocketMQ, Apache Kafka, Apache Pulsar, RabbitMQ, Redis.
• Rapidty extendsible meta such as Consul, Nacos, ETCD and Zookeeper.
• Guaranteed at-least-once delivery.
• Deliver events between multiple EventMesh deployments.
• Event schema management by catalog service.
• Powerful event orchestration by Serverless workflow engine.
• Powerful event filtering and transformation.
• Rapid, seamless scalability.
• Easy Function develop and framework integration.

Roadmap

Please go to the roadmap to get the release history and new features of Apache EventMesh.

Subprojects

• EventMesh-site: Apache official website resources for EventMesh.
• EventMesh-workflow: Serverless workflow runtime for event Orchestration on EventMesh.
• EventMesh-dashboard: Operation and maintenance console of EventMesh.
• EventMesh-catalog: Catalog service for event schema management using AsyncAPI.
• EventMesh-go: A go implementation for EventMesh runtime.

Quick start

This section of the guide will show you the steps to deploy EventMesh from Local, Docker, K8s.

This section guides the launch of EventMesh according to the default configuration, if you need more detailed EventMesh deployment steps, please visit the EventMesh official document.

Deployment Event Store

EventMesh supports multiple Event Stores, the default storage mode is standalone, and does not rely on other event stores as layers.

Run EventMesh Runtime locally

1. Download EventMesh

Download the latest version of the Binary Distribution from the EventMesh Download page and extract it:

wget https://dlcdn.apache.org/eventmesh/1.10.0/apache-eventmesh-1.10.0-bin.tar.gz
tar -xvzf apache-eventmesh-1.10.0-bin.tar.gz
cd apache-eventmesh-1.10.0

2. Run EventMesh

Execute the start.sh script to start the EventMesh Runtime server.

bash bin/start.sh

View the output log:

tail -n 50 -f logs/eventmesh.out

When the log output shows server state:RUNNING, it means EventMesh Runtime has started successfully.

You can stop the run with the following command:

bash bin/stop.sh

When the script prints shutdown server ok!, it means EventMesh Runtime has stopped.

Run EventMesh Runtime in Docker

1. Pull EventMesh Image

Use the following command line to download the latest version of EventMesh:

sudo docker pull apache/eventmesh:latest

2. Run and Manage EventMesh Container

Use the following command to start the EventMesh container:

sudo docker run -d --name eventmesh -p 10000:10000 -p 10105:10105 -p 10205:10205 -p 10106:10106 -t apache/eventmesh:latest

Enter the container:

sudo docker exec -it eventmesh /bin/bash

view the log:

cd logs
tail -n 50 -f eventmesh.out

Run EventMesh Runtime in Kubernetes

1. Deploy operator

Run the following commands(To delete a deployment, simply replace deploy with undeploy):

$ cd eventmesh-operator && make deploy

Run kubectl get pods 、kubectl get crd | grep eventmesh-operator.eventmeshto see the status of the deployed eventmesh-operator.

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
eventmesh-operator-59c59f4f7b-nmmlm   1/1     Running   0          20s

$ kubectl get crd | grep eventmesh-operator.eventmesh
connectors.eventmesh-operator.eventmesh   2024-01-10T02:40:27Z
runtimes.eventmesh-operator.eventmesh     2024-01-10T02:40:27Z

2. Deploy EventMesh Runtime

Execute the following command to deploy runtime, connector-rocketmq (To delete, simply replace create with delete):

$ make create

Run kubectl get pods to see if the deployment was successful.

NAME                                  READY   STATUS    RESTARTS   AGE
connector-rocketmq-0                  1/1     Running   0          9s
eventmesh-operator-59c59f4f7b-nmmlm   1/1     Running   0          3m12s
eventmesh-runtime-0-a-0               1/1     Running   0          15s

Contributing

Each contributor has played an important role in promoting the robust development of Apache EventMesh. We sincerely appreciate all contributors who have contributed code and documents.

• Contributing Guideline
• Good First Issues

Here is the List of Contributors, thank you all! :)

CNCF Landscape

Apache EventMesh enriches the CNCF Cloud Native Landscape.

License

Apache EventMesh is licensed under the Apache License, Version 2.0.

Community

WeChat Assistant	WeChat Public Account	Slack
		Join Slack Chat(Please open an issue if this link is expired)

Bi-weekly meeting : #Tencent meeting : 346-6926-0133

Bi-weekly meeting record : bilibili

Mailing List

Name	Description	Subscribe	Unsubscribe	Archive
Users	User discussion	Subscribe	Unsubscribe	Mail Archives
Development	Development discussion (Design Documents, Issues, etc.)	Subscribe	Unsubscribe	Mail Archives
Commits	Commits to related repositories	Subscribe	Unsubscribe	Mail Archives
Issues	Issues or PRs comments and reviews	Subscribe	Unsubscribe	Mail Archives

incubator-eventmesh's Issues

use kafka as storage

can we support using kafka as storage?

Does eventmesh support HTTP？Or any plan?

HTTP is one of the most commonly used protocols. Does EventMesh support HTTP? Or is there any plan to support it?

Suggest support rocketmq as store

can we provide client in c?

some system is developed by c, so it will be convenient to access if we can provide client in c.

The license of the script is recommended to be unified with the apache2 license

IMHO, It would be better to change the license to the apache2 license as other files.

How to build this project with Maven？

Event is built with gradle, can you provide a Maven version？

Does eventmesh do anything in security?

Data security is very important in finance. Does eventmesh do anything in security, such as access control, authentication?Or any plan?

some toString methods are single formated in eventmesh-common component

some toString() method not formated, this will lead print in a single without any seperator, inconvenient to view the log when analysis the problem.
such as:

HeartBeatInfo toString() method content is in the same line without any seperator
@Override public String toString() { return "HeartBeatInfo{" + "serviceId=" + serviceId + "instanceId=" + instanceId + "topicList=" + topicList + '}'; }
print result:
HeartBeatInfo{serviceId=serviceIdinstanceId=110topicList=[topic1, topic2]}

follow toString output maybe better
HeartBeatInfo{serviceId='serviceId', instanceId='110', topicList=[topic1, topic2]}

AccessMessage toString() method absence the body
public String toString() { return "AccessMessage{" + "topic='" + topic + '\'' + ", properties=" + properties + '}'; }

Does eventmesh support JMS ?

Does eventmesh support JMS ? How can we migrate to eventmesh ?

can eventmesh-emesher support rocketmq as a Independent component?

use eventmesh-emesher as a Independent component，support eventmesh-store or other mqs like rocketmq

any plan to support maven?

Hi All,

 any plan to support maven ?
 For our teams,  we use maven to manage our projects. If it can be build with maven, will give us more convenience。
 thanks

I look forward to your continued open source soon

希望后续的能够早些开源，饥渴难耐。

add tools module code

How to achieve high concurrency and low latency？

您好，请问DeFiBus在RocketMQ上增加的特性，是如何支撑其高并发、低延时特性的？

getPID method get current process id improve speed

getPID method get current process id waste time,because of call
ManagementFactory.getRuntimeMXBean() every time. call only once ok.

public static long getPID() {
String processName = java.lang.management.ManagementFactory.getRuntimeMXBean().getName();
if (processName != null && processName.length() > 0) {
try {
return Long.parseLong(processName.split("@")[0]);
} catch (Exception e) {
return 0;
}
}
return 0;
}

License problem on logback dependency

It seems that external dependency 'logback' is GPL license, I suggest you remove or replace this dependency

Any plan to use Nacos as a registration center?

As we know， Nacos supports the discovery, configuration and management of almost all mainstream types of "services"，is there any plan to use Nacos as a registration center?

proxy.properties contains ascii content

inconvenient to read
`
#\u6D88\u606F\u4E0B\u53D1\u5931\u8D25 \u6807\u8BB0\u5BA2\u6237\u7AEF\u9694\u79BB\u65F6\u95F4
proxy.server.tcp.pushFailIsolateTimeInMills=30000

#rebalance
proxy.server.tcp.RebalanceIntervalInMills=30000

#\u5BA2\u6237\u7AEFsession\u8FC7\u671F\u65F6\u95F4
proxy.server.session.expiredInMills=60000

#flow control,\u5206global\u7EA7\u522B\u548Csession\u7EA7\u522B
proxy.server.tcp.msgReqnumPerSecond=15000
proxy.server.session.upstreamBufferSize=20
proxy.server.session.downstreamUnackSize=100

#\u5168\u5C40\u8C03\u5EA6\u5668\u7EBF\u7A0B\u4E2A\u6570
`

can we provide client in python?

python is widely used in many project, can we provide client in python?

miti language support？

At present, only Java SDKs are seen in the project.
Are there plans to support other multiple language SDK? such as go or python?

Can we use redis as a storage for commitlog?

Can we use redis as a storage for commitlog,
in a distributed environment, we can share commitlog between differents brokers

Is there an open source plan for the unopen sourced part?

请问，像服务路由、代理、治理、平滑升级扩容等未开源部分，现在有开源计划了吗？

Hope to provide HTTP client

HTTP client is a convenient way for most people to use

In some cases, Producer has an uneven sending flow

There are two topics, T1 {Q0, Q1} and T2 {Q0, Q1}

The following timings exist

1. Counter = 0
2. Send T1, 0% Queue Sizeof T1 = 0, send to T1-Q0
3. Counter = 1
4. Send T2, 1% Queue Sizeof T2 = 1, send to T2-Q0
5. Counter = 2
6. Send T1, 2% Queue Sizeof T1 = 0, still send to T1-Q0
7. Counter = 3
8. Send T2, 3% Queue Sizeof T2 = 1, still send to T2-Q0
   ...

It can be seen that in step 6, for T1, two consecutive transmissions are sent to Q0,
In the same way, in step 8, for T2, two consecutive transmissions are sent to Q0, and the transmission flow is not balanced

The counter is global. For Topic T1, when counter% T1's Queue number = T1 send step size, there will be a problem of unbalanced flow

How to do event governance ？

Does eventmesh has any component to do event governance?
How to do event governance if there are ten thousthands or event millions of events ?

When ack arrives at the server prior to the callback ,may cause ack problem

When the message is pushed by the method SessionPusher.push() from emesher server to client ,client will send an ack msg to server, and then the server will do ack logics. At the same time, the server will execute the callback of pushed msg.

• Callback does: push the msg into unAcked map when msg has been pushed to client success.
• Ack does: execute msg's ack, remove from unAcked map.

The problem: when ack arrives to server prior to the callback, the server may ack failed, because the msg has not been pushed to unAcked map.

contains never throw exception in method

method has catch the exception
public void sendResponse(ChannelHandlerContext ctx, DefaultFullHttpResponse response) { ctx.writeAndFlush(response).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { if (!f.isSuccess()) { httpLogger.warn("send response to [{}] fail, will close this channel", RemotingHelper.parseChannelRemoteAddr(f.channel())); f.channel().close(); return; } } }); }

subscribe nearby may cause instance overload

As described in docs, message queue will be allocated to instances in same IDC even if there is only one consumer. When some unexpected accident happen and there is only one consumer alive, it will bear the load of whole IDC, which may cause this consumer overload.

The feature of "auto expand or shrink queue number" will keep queue number equals to the number of alive consumer, which enable us to know alive consumer number by evaluating queue number. If there is only one alive consumer, break the "nearby" rule and send message to not only message queue would be better to avoid overload.

fastjson version security issue

The version of fastjson is 1.2.61, this version has potential security risks

remove unnecessary code and support comment in emesher module

ProxyUtil.getLocalAddr() is a time-consuming operation, may affect the throughput of sending and receiving messages

ProxyUtil.getLocalAddr() is a time-consuming operation in the process of sending and receiving messages, which may affect the throughput. Maybe we can initialize at startup instead of calling this method every time in message transformation.

lack chinese introduction

I think it can help people get started faster if there are some chinese introduction.

Suggest DeFibus to export the supported features to the example

rt

wrong toString() method tag about PushMessageResponseBody

the toString() method about PushMessageResponseBody named to sendMessageResponseBody
should be pushMessageResponseBody
`
@OverRide
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("sendMessageResponseBody={")
.append("retCode=").append(retCode).append(",")
.append("retMsg=").append(retMsg).append(",")
.append("resTime=").append(DateFormatUtils.format(resTime, Constants.DATE_FORMAT)).append("}");
return sb.toString();
}

`

When do you plan to open source completely?

下述这些功能大概什么时候可以开源？ 有时间计划吗？

服务路由和定位：动态路由及定位(后续开源)
服务代理：TCP/HTTP及多语言的代理(后续开源)
服务治理：服务元数据的管理(后续开源)
平滑升级：平滑升级、平滑扩容(后续开源)

Do we support event filtering？

add emesher protocol description

some jar version like fastjson/guava has security leak

for example, fastjson, guava, slf4j-api, jackson, log4j-api

How can i find the discription of interaction protocol between SDK and EMesher?

Could not find cn.webank.defibus:defibus-client:1.0.0

travis ci is invalid

It seems the travis ci has some problem in eventmesh

Suggest to improve the directory structure

For example, if you want to extend multiple storage modules, each storage module should have a separate folder in the eventmesh-store directory, and abstract common parent classes

Wanted: who's using EventMesh

Who is Using EventMesh
First of all, thanks sincerely for constantly using and supporting DeFiBus. We will try our best to keep DeFiBus better, and keep growing community and eco-system.

The purpose of this issue
We’d like to listen to the community to make EventMesh better.
We want to attract more people to contribute to EventMesh.
We're willing to learn more EventMesh use scenarios in the field for better planning.
What we expect from you
Pls. submit a comment in this issue to include the following infomation:

your company, school or organization.
your city and country.
your contact info: blog, email, twitter (at least one).
for what business scenario do you use DeFiBus
You can refer to the following sample answer for the format:

• Orgnizatioin: WeBank
• Location: ShenZhen, China
• Contact: [email protected]
• Purpose: used as cornerstone in distributed service architecture webank switched years ago.
  Thanks again for your participation !
  EventMesh & DeFiBus Team

谁在使用 EventMesh
首先诚挚地感谢每一位持续关注并使用 DeFiBus的朋友。我们会持续投入，力图把 EventMesh变得更好，把 DeFiBus社区和生态变得更加繁荣。

此 Issue 的目的
聆听社区的声音，让 EventMesh变得更好
吸引更多的人参与贡献
更多的了解 EventMesh的实际使用场景，以方便下一步的规划
我们期待您能提供
在此提交一条评论, 评论内容包括:

您所在公司、学校或组织
您所在的城市、国家
您的联系方式: 微博、邮箱、微信 (至少一个)
您将 EventMesh用于哪些业务场景
您可以参考下面的样例来提供您的信息：

• 组织：微众银行
• 地点：**深圳
• 联系方式：[email protected]
• 场景：微众银行内部服务化改造
  再次感谢你的参与!!!
  EventMesh 团队

Does RPC call support any routing strategy ?

类似 路由上是树型分布的多节点. 一个请求需要发往全部或部分节点. 有些子节点网络是不通的.

是否支持通过路由转发到对应的节点.

Does EventMesh support MQTT?

IoT is popular nowadays，does eventmesh support MQTT ? Or any plan to support MQTT ?

Could we support AMQP protocol?

How to ensure each event is delivered "exactly once"?

eventmesh-store was implemented based on RocketMQ by default as described, does it support delivering each event "exactly once"? If so, How to do that ?

Problem in /eventmesh/docs

It seems that the documents in the docs directory only introduce DefiBus, and lack introductions to EventMesh

missing zh_Ch doc, eg readme, contributing guide, quickstart....

There are some duplicate import packages