saidsef/k8s-nifi-cluster

Apache NiFi cluster running in Kubernetes

Kubernetes NiFi Cluster

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

My goal is to show how to run Apache NiFi Cluster in Kubernetes

Prerequisites

• Kubernetes Cluster >= v1.23

Deployments

This will deploy Apache NiFi in a Cluster mode with extenal Apache Zookeeper managing ellections:

kubectl apply -k deployments/

NOTE: Remember to update Ingress hostname

This will create:

• 1x NiFi Namespace (all the items will be deployed here)
• 2x Apache NiFi Instances (can be increased)
• 1x Apache Zookeeper (accessible within the cluster only)
• 1x Secrets (basic auth username/passowrd: admin:admin)
• 1x Ingress (access endpoint)

ℹ️ Set Ingress hostname to valid hostname before enabling it in kustomization.yml Important: Remeber to update the default username/password.

Services

kubectl get all,ing

Contributing

We would ❤️ you to contribute by making a pull request.

Please read the official Contribution Guide for more information on how you can contribute.

Useful links:

• I Started NiFi. Now What?
• NiFi System Administrator’s Guide

k8s-nifi-cluster's Issues

java.io.EOFException

Not sure if this is an issue or just a problem with my deployment. When I run a "kubectl logs" on my nifi pods I keep seeing the following Exception. However, it is a little hard to complain because besides that things seem to be working. Wondering if you have seen (or are seeing) something similar, or if you might know how to resolve it? For what it is worth I really appreciate the effort you have put into this thus far.

2023-03-22 23:31:19,382 WARN [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from XXX.XXX.XXX.XXX due to java.io.EOFException
java.io.EOFException: null
at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:115)
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:150)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

fix ci release workflow

Release ci workflow seems to be failing due to the reorganisation of deployment directory. a quick fix might be searching for files with *.y[a]ml instead of all files in the directory.

Node adress gets 0.0.0.0:8080 and not pod ip

@saidsef Hello, thank you for your work. I tried to deploy Nifi Cluster in Kubernetes (Statefulset and also tried Deployment in kind value) and use same deployment but cluster node adress doesn't get the ip pod but 0.0.0.0:8080. I am able to have several nodes but they have same adress and I have this error in NiFi :
Node 0.0.0.0:8080 is unable to fulfill this request due to: Transaction XXX is already in progress.
Do you know where can it come from ?

Clarification Requested Re: Executing the Kubectl apply command

hello, I wanted to see if you could be clearer on how this code should be executed.
I have an active kubernetes cluster running.

I have downloaded the code from this github page.

I have revised the ingress.yml and updated the host with a node from my K8s cluster.
spec:
rules:

• host: $(host -t NS hkddbs.net)

But, I am uclear how to execute the kubectl apply command. Can you please provide any additional guidance?

Thank you