https://github.com/getindata/apache-nifi-kubernetes
https://github.com/getindata/apache-nifi-kubernetes
Last synced: 12 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/getindata/apache-nifi-kubernetes
- Owner: getindata
- Created: 2021-05-20T06:06:04.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2021-10-10T20:16:29.000Z (over 4 years ago)
- Last Synced: 2025-04-09T20:11:38.307Z (about 1 year ago)
- Language: Shell
- Size: 43 KB
- Stars: 20
- Watchers: 5
- Forks: 16
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Apache NiFi and NiFi Registry on Kubernetes
It is a repository with Helm charts for Apache NiFi and Apache NiFi Registry.
[Blog Post about NiFi on Kubernetes.](https://getindata.com/blog/)
- [Apache NiFi GitHub Repository](https://github.com/apache/nifi)
- [Apache NiFi Registry GitHub Repository](https://github.com/apache/nifi-registry)
## Note
This repo is constantly being improved.
## Prerequisites
You need the following components to use these Helm charts:
- Kubernetes cluster (1.15 and newer)
- Helm 3
Tested with the following Ingress:
- [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/)
Here we use featured [SSL Passthrough](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough).
## Tests
You can find CI pipelines for two CICD tools: GitLab CI and GitHub Actions.
- GitLab CI: .gitlab-ci.yml
- GitHub Actions: directory .github/workflows
## Installation
Create your values file or use the default one.
Apache NiFi:
```shell script
helm --namespace nifi nifi upgrade --install ./apache-nifi/chart -f ./apache-nifi/chart/values.yaml
```
Apache NiFi Registry:
```shell script
helm --namespace nifireg nifi upgrade --install ./apache-nifi-registry/chart -f ./apache-nifi-registry/chart/values.yaml
```
## Configuration - NiFi
The following table lists the configurable parameters of the Apache NiFi chart and the default values.
| Parameter | Description | Default |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------| ------------------------------- |
| **ReplicaCount** |
| `replicaCount` | Number of nifi nodes | `1` |
| **Image** |
| `image.repository` | nifi Image name | `apache/nifi` |
| `image.tag` | nifi Image tag | `1.11.4` |
| `image.pullPolicy` | nifi Image pull policy | `IfNotPresent` |
| `image.pullSecret` | nifi Image pull secret | `nil` |
| **SecurityContext** |
| `securityContext.runAsUser` | nifi Docker User | `1000` |
| `securityContext.fsGroup` | nifi Docker Group | `1000` |
| **sts** |
| `sts.podManagementPolicy` | Parallel podManagementPolicy | `Parallel` |
| `sts.AntiAffinity` | Affinity for pod assignment | `soft` |
| `sts.pod.annotations` | Pod template annotations | `security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000` |
| **secrets**
| `secrets` | Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. | `nil` |
| **configmaps**
| `configmaps` | Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. | `nil` |
| **nifi properties** |
| `properties.externalSecure` | externalSecure for when inbound SSL | `false` |
| `properties.isNode` | cluster node properties (only configure for cluster nodes) | `true` |
| `properties.httpPort` | web properties HTTP port | `8080` |
| `properties.httpsPort` | web properties HTTPS port | `null` |
| `properties.clusterPort` | cluster node port | `6007` |
| `properties.clusterSecure` | cluster nodes secure mode | `false` |
| `properties.needClientAuth` | nifi security client auth | `false` |
| `properties.provenanceStorage` | nifi provenance repository max storage size | `8 GB` |
| `properties.siteToSite.secure` | Site to Site properties Secure mode | `false` |
| `properties.siteToSite.port` | Site to Site properties Secure port | `10000` |
| `properties.siteToSite.authorizer` | | `managed-authorizer` |
| `properties.safetyValve` | Map of explicit 'property: value' pairs that overwrite other configuration | `nil` |
| **nifi user authentication** |
| `auth.ldap.enabled` | Enable User auth via ldap | `false` |
| `auth.ldap.host` | ldap hostname | `ldap://:` |
| `auth.ldap.searchBase` | ldap searchBase | `CN=Users,DC=example,DC=com` |
| `auth.ldap.searchFilter` | ldap searchFilter | `CN=john` |
| **postStart** |
| `postStart` | Include additional libraries in the Nifi containers by using the postStart handler | `nil` |
| **Headless Service** |
| `headless.type` | Type of the headless service for nifi | `ClusterIP` |
| `headless.annotations` | Headless Service annotations | `service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"`|
| **Ingress** |
| `ingress.enabled` | Enables Ingress | `false` |
| `ingress.annotations` | Ingress annotations | `{}` |
| `ingress.path` | Path to access frontend (See issue [#22](https://github.com/cetic/helm-nifi/issues/22)) | `/` |
| `ingress.hosts` | Ingress hosts | `[]` |
| `ingress.tls` | Ingress TLS configuration | `[]` |
| **Persistence** |
| `persistence.enabled` | Use persistent volume to store data | `false` |
| `persistence.storageClass` | Storage class name of PVCs (use the default type if unset) | `nil` |
| `persistence.accessMode` | ReadWriteOnce or ReadOnly | `[ReadWriteOnce]` |
| `persistence.dataStorage.size` | Size of persistent volume claim | `1Gi` |
| `persistence.flowfileRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.contentRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.provenanceRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.logStorage.size` | Size of persistent volume claim | `5Gi` |
| `persistence.existingClaim` | Use an existing PVC to persist data | `nil` |
| **jvmMemory** |
| `jvmMemory` | bootstrap jvm size | `2g` |
| **SideCar** |
| `sidecar.image` | Separate image for tailing each log separately | `ez123/alpine-tini` |
| `sidecar.tag` | Image tag | `latest` |
| **BusyBox** |
| `busybox.image` | Separate image for initContainer that verifies zookeeper is accessible | `busybox` |
| `busybox.tag` | Image tag | `latest` |
| **Resources** |
| `resources` | Pod resource requests and limits for logs | `{}` |
| **logResources** |
| `logresources.` | Pod resource requests and limits | `{}` |
| **nodeSelector** |
| `nodeSelector` | Node labels for pod assignment | `{}` |
| **terminationGracePeriodSeconds** |
| `terminationGracePeriodSeconds` | Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. | `60` |
| **tolerations** |
| `tolerations` | Tolerations for pod assignment | `[]` |
| **initContainers** |
| `initContainers` | Container definition that will be added to the pod as [initContainers](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#container-v1-core) | `[]` |
| **extraVolumes** |
| `extraVolumes` | Additional Volumes available within the pod (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volume-v1-core) for format) | `[]` |
| **extraVolumeMounts** |
| `extraVolumeMounts` | VolumeMounts for the nifi-server container (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumemount-v1-core) for details) | `[]` |
| **env** |
| `env` | Additional environment variables for the nifi-container (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#envvar-v1-core) for details) | `[]` |
| **extraContainers** |
| `extraContainers` | Additional container-specifications that should run within the pod (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#container-v1-core) for details) | `[]` |
## Configuration - NiFi Registry
The following table lists the configurable parameters of the Apache NiFi Registry chart and the default values.
| Parameter | Description | Default |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------| ------------------------------- |
| **ReplicaCount** |
| `replicaCount` | Number of nifi nodes | `1` |
| **Image** |
| `image.repository` | nifi Image name | `apache/nifi` |
| `image.tag` | nifi Image tag | `1.11.4` |
| `image.pullPolicy` | nifi Image pull policy | `IfNotPresent` |
| `image.pullSecret` | nifi Image pull secret | `nil` |
| **SecurityContext** |
| `securityContext.runAsUser` | nifi Docker User | `1000` |
| `securityContext.fsGroup` | nifi Docker Group | `1000` |
| **sts** |
| `sts.podManagementPolicy` | Parallel podManagementPolicy | `Parallel` |
| `sts.AntiAffinity` | Affinity for pod assignment | `soft` |
| `sts.pod.annotations` | Pod template annotations | `security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000` |
| **secrets**
| `secrets` | Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. | `nil` |
| **configmaps**
| `configmaps` | Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. | `nil` |
| **nifi properties** |
| `properties.externalSecure` | externalSecure for when inbound SSL | `false` |
| `properties.isNode` | cluster node properties (only configure for cluster nodes) | `true` |
| `properties.httpPort` | web properties HTTP port | `8080` |
| `properties.httpsPort` | web properties HTTPS port | `null` |
| `properties.clusterPort` | cluster node port | `6007` |
| `properties.clusterSecure` | cluster nodes secure mode | `false` |
| `properties.needClientAuth` | nifi security client auth | `false` |
| `properties.provenanceStorage` | nifi provenance repository max storage size | `8 GB` |
| `properties.siteToSite.secure` | Site to Site properties Secure mode | `false` |
| `properties.siteToSite.port` | Site to Site properties Secure port | `10000` |
| `properties.siteToSite.authorizer` | | `managed-authorizer` |
| `properties.safetyValve` | Map of explicit 'property: value' pairs that overwrite other configuration | `nil` |
| **nifi user authentication** |
| `auth.ldap.enabled` | Enable User auth via ldap | `false` |
| `auth.ldap.host` | ldap hostname | `ldap://:` |
| `auth.ldap.searchBase` | ldap searchBase | `CN=Users,DC=example,DC=com` |
| `auth.ldap.searchFilter` | ldap searchFilter | `CN=john` |
| **postStart** |
| `postStart` | Include additional libraries in the Nifi containers by using the postStart handler | `nil` |
| **Headless Service** |
| `headless.type` | Type of the headless service for nifi | `ClusterIP` |
| `headless.annotations` | Headless Service annotations | `service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"`|
| **Ingress** |
| `ingress.enabled` | Enables Ingress | `false` |
| `ingress.annotations` | Ingress annotations | `{}` |
| `ingress.path` | Path to access frontend (See issue [#22](https://github.com/cetic/helm-nifi/issues/22)) | `/` |
| `ingress.hosts` | Ingress hosts | `[]` |
| `ingress.tls` | Ingress TLS configuration | `[]` |
| **Persistence** |
| `persistence.enabled` | Use persistent volume to store data | `false` |
| `persistence.storageClass` | Storage class name of PVCs (use the default type if unset) | `nil` |
| `persistence.accessMode` | ReadWriteOnce or ReadOnly | `[ReadWriteOnce]` |
| `persistence.dataStorage.size` | Size of persistent volume claim | `1Gi` |
| `persistence.flowfileRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.contentRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.provenanceRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.logStorage.size` | Size of persistent volume claim | `5Gi` |
| `persistence.existingClaim` | Use an existing PVC to persist data | `nil` |
| **jvmMemory** |
| `jvmMemory` | bootstrap jvm size | `2g` |
| **SideCar** |
| `sidecar.image` | Separate image for tailing each log separately | `ez123/alpine-tini` |
| `sidecar.tag` | Image tag | `latest` |
| **BusyBox** |
| `busybox.image` | Separate image for initContainer that verifies zookeeper is accessible | `busybox` |
| `busybox.tag` | Image tag | `latest` |
| **Resources** |
| `resources` | Pod resource requests and limits for logs | `{}` |
| **logResources** |
| `logresources.` | Pod resource requests and limits | `{}` |
| **nodeSelector** |
| `nodeSelector` | Node labels for pod assignment | `{}` |
| **terminationGracePeriodSeconds** |
| `terminationGracePeriodSeconds` | Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. | `60` |
| **tolerations** |
| `tolerations` | Tolerations for pod assignment | `[]` |
| **initContainers** |
| `initContainers` | Container definition that will be added to the pod as [initContainers](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#container-v1-core) | `[]` |
| **extraVolumes** |
| `extraVolumes` | Additional Volumes available within the pod (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volume-v1-core) for format) | `[]` |
| **extraVolumeMounts** |
| `extraVolumeMounts` | VolumeMounts for the nifi-server container (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumemount-v1-core) for details) | `[]` |
| **env** |
| `env` | Additional environment variables for the nifi-container (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#envvar-v1-core) for details) | `[]` |
| **extraContainers** |
| `extraContainers` | Additional container-specifications that should run within the pod (see [spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#container-v1-core) for details) | `[]` |
## Credits
Inspired from https://github.com/cetic/helm-nifi