Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/AlexsJones/kubernetes-nifi-cluster
Apache Nifi cluster running in kubernetes
https://github.com/AlexsJones/kubernetes-nifi-cluster
kubernetes kubernetes-nifi-cluster kubernetes-statefulset nifi persistent-volume-claims zookeeper
Last synced: 4 months ago
JSON representation
Apache Nifi cluster running in kubernetes
- Host: GitHub
- URL: https://github.com/AlexsJones/kubernetes-nifi-cluster
- Owner: AlexsJones
- License: mit
- Created: 2017-11-20T18:19:43.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-09-03T08:50:57.000Z (over 5 years ago)
- Last Synced: 2024-10-27T23:54:21.664Z (4 months ago)
- Topics: kubernetes, kubernetes-nifi-cluster, kubernetes-statefulset, nifi, persistent-volume-claims, zookeeper
- Language: Shell
- Size: 31.3 KB
- Stars: 84
- Watchers: 8
- Forks: 21
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ccamel - AlexsJones/kubernetes-nifi-cluster - Apache Nifi cluster running in kubernetes (Shell)
README
# _deprecated_ kubernetes-nifi-cluster _please use https://github.com/AlexsJones/nifi_
A nifi cluster running in kubernetes
## Helm chart now available
This repo is now being deprecated in favour of a Helm compatible [chart](https://github.com/AlexsJones/nifi)
_I am going to appeal to add it into helm/charts as soon as my dependent zookeeper chart is added_
Thank you!
### Prerequisites
- This example is using Google Cloud Platform persistent volumes for its backing store (easy to convert to AWS).
- There is no longer a requirement to install [vortex](https://github.com/AlexsJones/vortex) for interpolation.
- For setting up the environment run:
```
docker run -v $PWD:/tmp tibbar/vortex:v1 -template /tmp/templates -output /tmp/deployment -varpath /tmp/environments/default.yaml
```
This will generate your default environment
- Requires [zookeeper](https://github.com/AlexsJones/kubernetes-zookeeper-cluster)
```
# Assumes you've checked it out next to the nifi cluster...
cd kubernetes-zookeeper-cluster
docker run -v $PWD:/tmp tibbar/vortex:v1 -template /tmp/templates -output /tmp/deployment -varpath /tmp/environments/small.yaml
kubectl create -f deployment/
cd ../kubernetes-nifi-cluster
```
Now zookeeper is setup with three nodes on the zk namespace you are ready!
## Deploy Nifi
_Make sure zk is running to avoid headaches_
```
# Please create the nifi namespace first
# Ensure you have run docker vortex build step mentioned above^
kubectl create ns nifi
kubectl create -f deployment/
```
Once running you should see...
```
kubectl get pods -n nifi
NAME READY STATUS RESTARTS AGE
nifi-0 1/1 Running 0 25m
nifi-1 1/1 Running 0 25m
nifi-2 1/1 Running 0 25m
kubectl get pvc -n nifi
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
contentrepository-nifi-0 Bound pvc-c00b39d5-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
contentrepository-nifi-1 Bound pvc-c0116c25-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
contentrepository-nifi-2 Bound pvc-c019d7ee-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
databaserepository-nifi-0 Bound pvc-c00a3682-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
databaserepository-nifi-1 Bound pvc-c00f87a8-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
databaserepository-nifi-2 Bound pvc-c017dbe4-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
flowfilerepository-nifi-0 Bound pvc-c0096aac-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
flowfilerepository-nifi-1 Bound pvc-c00df6bb-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
flowfilerepository-nifi-2 Bound pvc-c016020d-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
provenancerepository-nifi-0 Bound pvc-c008b6bd-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
provenancerepository-nifi-1 Bound pvc-c0132c86-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
provenancerepository-nifi-2 Bound pvc-c01aec6b-4710-11e9-b1b0-42010a800055 5Gi RWO standard 1d
```
---
## UI
Port forward to the UX:
```
kubectl port-forward nifi-0 8080:8080 -n nifi
```
`http://:8080/nifi/`
## Scaling
`kubectl scale --replicas sts/nifi`
## Configuration
The `environments/` folder can have new files added and used with `build_environment.sh ` and those values interpolated
For more informatino on this please see [here](https://github.com/AlexsJones/vortex/blob/master/README.md)
### Production readiness
A checklist for consideration I would think about before you want to run this in prod
- Use an identity aware proxy infront of the load balancer
- Recovery procedure from the PVC's
- Scaling, liveliness & readiness probes