https://github.com/tomarv2/nifi
NiFi deployment as StatefulSet & CICD
https://github.com/tomarv2/nifi
k8s kubernetes nifi nifi-api nifi-cluster nifi-docker statefulsets
Last synced: about 2 months ago
JSON representation
NiFi deployment as StatefulSet & CICD
- Host: GitHub
- URL: https://github.com/tomarv2/nifi
- Owner: tomarv2
- License: apache-2.0
- Created: 2019-04-29T23:02:26.000Z (over 6 years ago)
- Default Branch: main
- Last Pushed: 2025-01-15T17:32:25.000Z (9 months ago)
- Last Synced: 2025-03-22T15:49:52.122Z (7 months ago)
- Topics: k8s, kubernetes, nifi, nifi-api, nifi-cluster, nifi-docker, statefulsets
- Language: Python
- Homepage:
- Size: 14.5 MB
- Stars: 4
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## NiFi deployment & CICD
Deploy a NiFi cluster as StatefulSet in k8s and continuous deployment of applications.
***
### Covering two use cases:
- Setting up NiFi in minikube (this is what we provide to developers as a playground).
- Production ready deployment.
***
When we initially started deploying NiFi it was version on 1.5, k8s support for NiFi was not that great.
Things have changed a lot since then.
***
### This repo addresses two main concerns:
- How to Deploy NiFi as a StatefulSet application
- How to do CICD
***
### Pre-reqs
- k8s cluster
- Zookeeper to maintain NiFi state outside cluster
- Persistent disk
- Prometheus
- Two good options for exporting nifi metrics in prometheus supported format:
- https://github.com/msiedlarek/nifi_exporter - haven't tried it as yet
- https://github.com/mkjoerg/nifi-prometheus-reporter - tested & in use
***
### Repo structure
:wave: Repo is divided into two parts: `k8s stateful deployment` and `building NiFi docker image`.
- `_kube`: k8s cluster deployment related files
- `_kube/config`: Properties files
- `_kube/templates`: templates (in xml format)
- `application/nifi_setup_1.11.4_stateful/build/deploy`: All scripts for building the NiFi docker image reside
- `application/nifi_setup_1.11.4_stateful/build/custom_processors`: All custom processors jar files
***
### CICD process
- As NiFi is a stateful app, had to make some adjustments to our rollout, with every run:
#### Delete Statefulset app
```
sh "kubectl delete --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true statefulsets ${env.serviceName} --cascade=false"
```
***
#### Delete Statefulset pods
```
sh "kubectl delete --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true pods -l cluster=${env.serviceName} -n ${env.NameSpace} --force --grace-period=0"
```
***
#### Delete PVC
```
sh "kubectl delete --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true pvc -l cluster=${env.serviceName} -n ${env.NameSpace} "
```
***
#### Deploy Statefulset
```
steps {
script {
try {
withCredentials([usernamePassword(credentialsId: 'k8s_cluster_pwd_qa', passwordVariable: 'k8s_pwd', usernameVariable: 'k8s_user')]) {
sh "kubectl create --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true -f _kube/qa/statefulset.yaml"
sh("eval \$(kubectl describe svc ${env.serviceName} --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true |grep -i nodeport |grep -i web |grep -v hook |awk {'print \$3'}|cut -f1 -d '/' > NodePort)")
script {
env.NODEPORT = readFile('NodePort')
sh "cat NodePort"
}
}
}
catch (exc) {sh 'echo statefulset does not exist...'}
}
}
```
***
- Deploy NiFi Template & Deploy Monitoring
***
### How to run
```
- python deploy_nifi.py
- python deploy_nifi.py http://nifi.services.demo.com:80 http://varun.demo.com/projects/raw/templates nifi-template.xml application_name
```
***
### Note:
- We are also using a headless service
- We expose nifi metrics in prometheus format
- Update `_kube/statefulset.yaml` with right information
- You will notice the `annotations` in `Statefulset.yaml` file, we are exporting metrics and logs to Datadog
- `nifi_setup_1.11.4_stateful/build/deploy/nifi.sh` points to custom `logback.xml` file
- Custom `logback.xml` to handle log in log issue with nifi logs(not sure if this is resolved in the latest version)
- `statefulset.yaml` is highly commented to work in minikube or in prod cluster with minimal changes
- Currently it works with `SecurityContext` will be changing it to work with PSP
***
#### Tip
:wave: The best way to use k8s related files just search and replace `sample-sync` with the required name and you are ready to go.
***
### Minikube setup
```
demo$ minikube service list
|----------------------|---------------------------|--------------|--------------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|----------------------|---------------------------|--------------|--------------------------------|
| default | kubernetes | No node port |
| devops | sample-sync | | http://192.168.64.2:32403 |
| | | | http://192.168.64.2:31162 |
| | | | http://192.168.64.2:32383 |
| | | | http://192.168.64.2:32412 |
| devops | sample-sync-headless | No node port |
| kube-system | kube-dns | No node port |
| kubernetes-dashboard | dashboard-metrics-scraper | No node port |
| kubernetes-dashboard | kubernetes-dashboard | No node port |
|----------------------|---------------------------|--------------|--------------------------------|
```
### Enable ingress on minikube
```
demo$ minikube addons enable ingress
The 'ingress' addon is enabled
```
### Headless service
:wave: A Headless Service is a service when you don’t need load-balancing and a single Service IP. Instead of load-balancing it will return the IPs of the attached Pod. Headless Services do not have a Cluster IP associated. Request will not be proxied by kube-proxy, instead NiFi will handle the service discovery.
#### Note:
:wave: If you find any issues or wants to see anything specific please open an issue.