polyzos/pulsar-geo-replication-demo

Pulsar Geo-Replication and ActiveMQ Migration

The purpose of this repository is 2-fold:

1. Demonstrate how we can migrate our data from ActiveMQ to Pulsar
2. Demonstrate how we can provide Disaster Recovery solutions using Geo-Replication:
   • Demonstrate using multiple pulsar cluster on the same k8s cluster
   • Demonstrate using multiple pulsar cluster on different k8s clusters
   • How we can seemlessy fallback to another cluster if we lose a regional cluster

You can find instructions to test geo-replication both with open-source pulsar as well as the SN Platform

Folders

1. jars contains the activemq connector nar
2. /infra contains terraform resources for setting up the necessary pulsar components easily on the different clusters
3. /k8s contains k8s manifests for deploying ActiveMQ on kubernetes as well as manifest for deploying pulsar and the sn platform

Setup & Instructions

1. List eu-central EKS clusters

aws eks list-clusters --region eu-central-1

3. Set kubeconfig

aws eks --region eu-central-1 update-kubeconfig \
  --name <cluster_name>

4. Create a namespace

kubectl create ns pulsar

5. Setup 3 pulsar clusters

helm install \
    --values k8s/values.yaml \
    --set initialize=true \
    pulsar-west apache/pulsar
    
helm install \
    --values k8s/values.yaml \
    --set initialize=true \
    pulsar-east apache/pulsar
    
helm install \
    --values k8s/values.yaml \
    --set initialize=true \
    pulsar-central apache/pulsar

6. Get 3 terminals for Pulsar admin

kubectl exec -it pulsar-west-toolset-0 -n pulsar -- bash
kubectl exec -it pulsar-east-toolset-0 -n pulsar -- bash
kubectl exec -it pulsar-central-toolset-0 -n pulsar -- bash

7. List clusters

bin/pulsar-admin clusters list

8. Get the clusters

bin/pulsar-admin clusters get pulsar-west
bin/pulsar-admin clusters get pulsar-east
bin/pulsar-admin clusters get pulsar-central

9. Register east and central cluster to west

bin/pulsar-admin clusters create \
  --broker-url http://pulsar-east-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-east-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-east
  
bin/pulsar-admin clusters create \
  --broker-url http://pulsar-central-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-central-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-central

9. Register west and central cluster to east

bin/pulsar-admin clusters create \
  --broker-url http://pulsar-west-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-west-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-west
  
bin/pulsar-admin clusters create \
  --broker-url http://pulsar-central-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-central-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-central

10. Register west and east cluster to central

bin/pulsar-admin clusters create \
  --broker-url http://pulsar-west-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-west-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-west
  
bin/pulsar-admin clusters create \
  --broker-url http://pulsar-east-broker.pulsar.svc.cluster.local:8080/ \
  --url pulsar://pulsar-east-broker.pulsar.svc.cluster.local:6650/ \
  pulsar-east

11. Create tenants and namespaces on all 3 clusters

bin/pulsar-admin tenants create testt --allowed-clusters pulsar-east,pulsar-west,pulsar-central
bin/pulsar-admin namespaces create testt/testns --clusters pulsar-east,pulsar-west,pulsar-central

12. Run consumers on east and central clusters and the producer on the west cluster

bin/pulsar-client consume -s "sub" testt/testns/t1 -n 0

bin/pulsar-client produce  testt/testns/t1  --messages "hello from the west cluster" -n 10

kubectl port-forward service/pulsar-central-proxy 8080:8080 6650:6650 -n pulsar
kubectl port-forward service/pulsar-east-proxy 8081:8080 6651:6650 -n pulsar
kubectl port-forward service/pulsar-west-proxy 6652:6650 -n pulsar

bin/pulsar zookeeper-shell

Streamnative Platform

1. Set kubeconfig for central

aws eks list-clusters --region eu-central-1

aws eks --region eu-central-1 update-kubeconfig \
  --name sn-tarpon-dev-eu-central-1

2. Set kubeconfig for west

aws eks list-clusters --region eu-west-1

aws eks --region eu-west-1 update-kubeconfig \
  --name sn-bee-dev-eu-west-1

4. Install SN Platform

helm upgrade --install vault-operator banzaicloud-stable/vault-operator -n pulsar
helm upgrade --install cert-manager jetstack/cert-manager --set installCRDs=true -n pulsar
helm upgrade --install pulsar-operator streamnative/pulsar-operator -n pulsar
helm upgrade --install function-mesh streamnative/function-mesh-operator -n pulsar 

helm install -f k8s/sn/values.yaml pulsar-central streamnative/sn-platform --set initialize=true -n pulsar
helm install -f k8s/sn/values.yaml pulsar-west streamnative/sn-platform --set initialize=true -n pulsar

5. Get the secret

kubectl get secret pulsar-central-sn-platform-vault-console-admin-passwd \
  -o=jsonpath='{.data.password}' \
   -n pulsar | base64 --decode; echo
   
kubectl get secret pulsar-west-sn-platform-vault-console-admin-passwd \
  -o=jsonpath='{.data.password}' \
   -n pulsar | base64 --decode; echo
   
Root token: 9IjZ7UBeovB6

6. Get west cluster info

# Get the root token
kubectl get secret pulsar-west-sn-platform-vault-secret-env-injection \
  -o=jsonpath='{.data.brokerClientAuthenticationParameters}' \
  -n pulsar | base64 --decode; echo

Yzc2NzQzMjItNDI5Mi01OWFlLWJiYWEtMjU0M2U4YWZhZDQ1OmVjMWZiODNmLTA0NWEtOTYxNi1hOTMwLTdkZjI5ZjEzOTQyNAo=

# Go into the toolset pod 
kubectl exec -it pulsar-west-sn-platform-toolset-0 -n pulsar -- bash

# List the clusters
pulsar-admin clusters list

pulsar-west-sn-platform

# Get the cluster info
pulsar-admin clusters get pulsar-west-sn-platform

6. Get central cluster info

# Get the root token
kubectl get secret pulsar-central-sn-platform-vault-secret-env-injection \
  -o=jsonpath='{.data.brokerClientAuthenticationParameters}' \
  -n pulsar | base64 --decode; echo

NDIwYzhhNTAtMzc4OS0yODE5LWQyZjItZGM5Yjc3NzQzYmJiOjc0NmRmN2M3LWJhYzAtYTM3YS1hMzgxLWMwODE1ZDc0YWQxNAo=

# Go into the toolset pod 
kubectl exec -it pulsar-central-sn-platform-toolset-0 -n pulsar -- bash

# List the clusters
pulsar-admin clusters list

pulsar-central-sn-platform

# Get the cluster info
pulsar-admin clusters get pulsar-central-sn-platform

7. Register the west to central

bin/pulsar-admin clusters create \
  --broker-url pulsar://<west-url>:6650 \
  --url http://<west-url>:8080 \
  --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
  --auth-parameters token:<west-token>=\
  pulsar-west-sn-platform

8. Register the central to west

bin/pulsar-admin clusters create \
  --broker-url pulsar://<central-url>:6650 \
  --url http://<central-url>:8080 \
  --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
  --auth-parameters token:<central-token>\
  pulsar-central-sn-platform

9. Create resources

bin/pulsar-admin tenants create testt --allowed-clusters pulsar-west-sn-platform,pulsar-central-sn-platform
bin/pulsar-admin namespaces create testt/testns --clusters pulsar-west-sn-platform,pulsar-central-sn-platform