zeebe-hazelcast-exporter

Export records from Zeebe to Hazelcast. Hazelcast is an in-memory data grid which is used as a transport layer.

The records are transformed into Protobuf and added to one ringbuffer. The ringbuffer has a fixed capacity and will override the oldest entries when the capacity is reached.

Multiple applications can read from the ringbuffer. The application itself controls where to read from by proving a sequence number. Every application can read from a different sequence.

The Java and C# connector modules provide a convenient way to read the records from the ringbuffer.

Usage

Java Application

Add the Maven dependency to your pom.xml

<dependency>
	<groupId>io.zeebe.hazelcast</groupId>
	<artifactId>zeebe-hazelcast-connector</artifactId>
	<version>%{VERSION}</version>
</dependency>

Connect to Hazelcast and register a listener

ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().addAddress("127.0.0.1:5701");
HazelcastInstance hz = HazelcastClient.newHazelcastClient(clientConfig);

final ZeebeHazelcast zeebeHazelcast = ZeebeHazelcast.newBuilder(hz)
    .addWorkflowInstanceListener(workflowInstance -> { ... })
    .readFrom(sequence) / .readFromHead() / .readFromTail()
    .build();

// ...

zeebeHazelcast.close();

C# Application

Add the nuget package zeebe hazelcast connector(???) to your project.

Example usage:

    // Start the Hazelcast Client and connect to an already running Hazelcast Cluster on 127.0.0.1
    var hz = HazelcastClient.NewHazelcastClient();
    // Get a Topic called "zeebe-deployments"
    var topic = hz.GetTopic<byte[]>("zeebe-deployments");

     // Add a Listener to the Topic
    DeploymentListener.Consumer consumer = (record) => Console.WriteLine(record.ToString());
    topic.AddMessageListener(new DeploymentListener(consumer));

Install

Docker

A docker image is published to DockerHub that is based on the Zeebe image and includes the Hazelcast exporter (the exporter is enabled by default).

docker pull camunda/zeebe-with-hazelcast-exporter:latest

For a local setup, the repository contains a docker-compose file. It starts a Zeebe broker with the Hazelcast exporter. The version of the exporter is defined in the .env file.

mvn clean install -DskipTests
cd docker
docker-compose up

Manual

Download the latest Zeebe distribution (zeebe-distribution-%{VERSION}.tar.gz )

Copy the exporter JAR into the broker folder ~/zeebe-broker-%{VERSION}/exporters.

cp exporter/target/zeebe-hazelcast-exporter-%{VERSION}-jar-with-dependencies.jar ~/zeebe-broker-%{VERSION}/exporters/

Add the exporter to the broker configuration ~/zeebe-broker-%{VERSION}/config/application.yaml:

zeebe:
  broker:  
    exporters:
      hazelcast:
        className: io.zeebe.hazelcast.exporter.HazelcastExporter
        jarPath: exporters/zeebe-hazelcast-exporter-%{VERSION}-jar-with-dependencies.jar

For broker version < 0.23.0-alpha2 ~/zeebe-broker-%{VERSION}/conf/zeebe.cfg.toml:

[[exporters]]
id = "hazelcast"
className = "io.zeebe.hazelcast.exporter.HazelcastExporter"
jarPath = "exporters/zeebe-hazelcast-exporter-%{VERSION}-jar-with-dependencies.jar"

Start the broker ~/zeebe-broker-%{VERSION}/bin/broker

Configuration

In the Zeebe configuration file, you can change

the Hazelcast port
the value and record types which are exported
the ringbuffer's name
the ringbuffer's capacity
the ringbuffer's time-to-live
the record serialization format

Default values:

zeebe:
  broker:
    exporters:
      hazelcast:
        className: io.zeebe.hazelcast.exporter.HazelcastExporter
        jarPath: exporters/zeebe-hazelcast-exporter.jar
	args:
	  # Hazelcast port
    	  port = 5701
    
          # comma separated list of io.zeebe.protocol.record.ValueType to export or empty to export all types 
          enabledValueTypes = ""
    
          # comma separated list of io.zeebe.protocol.record.RecordType to export or empty to export all types
          enabledRecordTypes = ""
        
          # Hazelcast ringbuffer's name
          name = "zeebe"
    
          # Hazelcast ringbuffer's capacity
          capacity = 10000 

          # Hazelcast ringbuffer's time-to-live in seconds. Don't remove the records until reaching the capacity by setting it to 0.  
          timeToLiveInSeconds = 0

          # record serialization format: [protobuf|json]
          format = "protobuf"

The values can be overridden by environment variables with the same name and a ZEEBE_HAZELCAST_ prefix (e.g. ZEEBE_HAZELCAST_PORT).

Connect to an External/Remote Hazelcast Cluster

By default, the exporter creates an in-memory Hazelcast instance and publishes the records to it. But it can also be configured to export the records to a remote/external Hazecast instance by setting the argument remoteAddress or the environment variable ZEEBE_HAZELCAST_REMOTE_ADDRESS to the address of the remote Hazelcast instance.

Full docker-compose.yml with external Hazelcast

version: "2"

networks:
  zeebe_network:
    driver: bridge

services:
  zeebe:
    container_name: zeebe_broker
    image: camunda/zeebe:0.23.1
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_HAZELCAST_REMOTE_ADDRESS=hazelcast:5701
    ports:
      - "26500:26500"
      - "9600:9600"
    volumes:
      - ../exporter/target/zeebe-hazelcast-exporter-${EXPORTER_VERSION}-jar-with-dependencies.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar
      - ./application.yaml:/usr/local/zeebe/config/application.yaml
    networks:
      - zeebe_network
    depends_on:
      - hazelcast

  hazelcast:
    container_name: hazelcast
    image: hazelcast/hazelcast:4.0.1
    ports:
      - "5701:5701"
    environment:
      - JAVA_OPTS="-Dhazelcast.local.publicAddress=hazelcast:5701"
    networks:
      - zeebe_network
      
  hazelcast-management:
    container_name: hazelcast-management
    image: hazelcast/management-center:4.0.1
    ports:
      - "8083:8080"
    networks:
      - zeebe_network
    depends_on:
      - hazelcast

Build it from Source

The exporter and the Java connector can be built with Maven

mvn clean install

Code of Conduct

This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].

hatzhang / zeebe-hazelcast-exporter Goto Github PK