{"id":26575961,"url":"https://github.com/tomarv2/nifi","last_synced_at":"2025-10-08T15:06:38.024Z","repository":{"id":44611117,"uuid":"184156043","full_name":"tomarv2/nifi","owner":"tomarv2","description":"NiFi deployment as StatefulSet \u0026 CICD","archived":false,"fork":false,"pushed_at":"2025-01-15T17:32:25.000Z","size":15157,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-08T23:37:01.918Z","etag":null,"topics":["k8s","kubernetes","nifi","nifi-api","nifi-cluster","nifi-docker","statefulsets"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomarv2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-29T23:02:26.000Z","updated_at":"2025-01-15T17:32:20.000Z","dependencies_parsed_at":"2025-01-15T18:50:37.200Z","dependency_job_id":"8890f2ce-7900-4251-81da-45ffe98bf245","html_url":"https://github.com/tomarv2/nifi","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tomarv2/nifi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarv2%2Fnifi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarv2%2Fnifi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarv2%2Fnifi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarv2%2Fnifi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomarv2","download_url":"https://codeload.github.com/tomarv2/nifi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomarv2%2Fnifi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278963977,"owners_count":26076573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s","kubernetes","nifi","nifi-api","nifi-cluster","nifi-docker","statefulsets"],"created_at":"2025-03-23T02:35:53.821Z","updated_at":"2025-10-08T15:06:37.995Z","avatar_url":"https://github.com/tomarv2.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.apache.org/licenses/LICENSE-2.0\" alt=\"GitHub tag\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/license/tomarv2/nifi\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/tomarv2/nifi/tags\" alt=\"GitHub tag\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/v/tag/tomarv2/nifi\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://stackoverflow.com/users/6679867/tomarv2\" alt=\"Stack Exchange reputation\"\u003e\n        \u003cimg src=\"https://img.shields.io/stackexchange/stackoverflow/r/6679867\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://discord.gg/XH975bzN\" alt=\"chat on Discord\"\u003e\n        \u003cimg src=\"https://img.shields.io/discord/813961944443912223?logo=discord\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://twitter.com/intent/follow?screen_name=varuntomar2019\" alt=\"follow on Twitter\"\u003e\n        \u003cimg src=\"https://img.shields.io/twitter/follow/varuntomar2019?style=social\u0026logo=twitter\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## NiFi deployment \u0026 CICD\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://files.gitter.im/tomarv2/gEzT/nifi_jenkins.png\"\u003e\n\u003c/p\u003e\n\nDeploy a NiFi cluster as StatefulSet in k8s and continuous deployment of applications.\n\n***\n### Covering two use cases:\n\n- Setting up NiFi in minikube (this is what we provide to developers as a playground).\n- Production ready deployment.\n\n***\nWhen we initially started deploying NiFi it was version on 1.5, k8s support for NiFi was not that great.\nThings have changed a lot since then.\n\n***\n### This repo addresses two main concerns:\n\n-  How to Deploy NiFi as a StatefulSet application\n-  How to do CICD\n\n***\n### Pre-reqs\n- k8s cluster\n- Zookeeper to maintain NiFi state outside cluster\n- Persistent disk\n- Prometheus \n  - Two good options for exporting nifi metrics in prometheus supported format:\n    - https://github.com/msiedlarek/nifi_exporter - haven't tried it as yet\n    - https://github.com/mkjoerg/nifi-prometheus-reporter - tested \u0026 in use\n***\n### Repo structure\n\n:wave: Repo is divided into two parts: `k8s stateful deployment` and `building NiFi docker image`.\n\n- `_kube`: k8s cluster deployment related files\n\n- `_kube/config`: Properties files\n\n- `_kube/templates`: templates (in xml format)\n\n- `application/nifi_setup_1.11.4_stateful/build/deploy`: All scripts for building the NiFi docker image reside \n\n- `application/nifi_setup_1.11.4_stateful/build/custom_processors`: All custom processors jar files\n\n***\n### CICD process\n\n- As NiFi is a stateful app, had to make some adjustments to our rollout, with every run:\n\n#### Delete Statefulset app\n\n```\nsh \"kubectl delete --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true statefulsets ${env.serviceName} --cascade=false\"\n```\n\n***\n#### Delete Statefulset pods\n\n```\nsh \"kubectl delete --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd}  --insecure-skip-tls-verify=true pods -l cluster=${env.serviceName} -n ${env.NameSpace} --force --grace-period=0\"\n```\n\n***\n#### Delete PVC\n\n```\nsh \"kubectl delete --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd}  --insecure-skip-tls-verify=true pvc -l cluster=${env.serviceName} -n ${env.NameSpace} \"\n```\n\n***\n#### Deploy Statefulset\n\n```\nsteps {\n    script {\n        try {\n            withCredentials([usernamePassword(credentialsId: 'k8s_cluster_pwd_qa', passwordVariable: 'k8s_pwd', usernameVariable: 'k8s_user')]) {\n                sh \"kubectl create --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true -f _kube/qa/statefulset.yaml\"\n                sh(\"eval \\$(kubectl describe svc ${env.serviceName} --namespace=${env.NameSpace} --server='https://qak8s-master.demo.com' --username=${k8s_user} --password=${k8s_pwd} --insecure-skip-tls-verify=true |grep -i nodeport |grep -i web |grep -v hook |awk {'print \\$3'}|cut -f1 -d '/' \u003e NodePort)\")\n                script {\n                    env.NODEPORT = readFile('NodePort')\n                    sh \"cat NodePort\"\n                }\n            }\n        }\n        catch (exc) {sh 'echo statefulset does not exist...'}\n    }\n}\n```\n\n***\n- Deploy NiFi Template \u0026 Deploy Monitoring\n\n***\n### How to run\n\n```\n- python deploy_nifi.py \u003cnifi_url\u003e \u003crepo location of templates\u003e \u003ctemplate_name\u003e\u003cproject_name\u003e\n    \n- python deploy_nifi.py http://nifi.services.demo.com:80 http://varun.demo.com/projects/raw/templates nifi-template.xml application_name\n```\n\n***\n### Note:\n\n- We are also using a headless service\n\n- We expose nifi metrics in prometheus format\n\n- Update  `_kube/statefulset.yaml` with right information\n\n- You will notice the `annotations` in `Statefulset.yaml` file, we are exporting metrics and logs to Datadog\n\n- `nifi_setup_1.11.4_stateful/build/deploy/nifi.sh` points to custom `logback.xml` file\n\n- Custom `logback.xml` to handle log in log issue with nifi logs(not sure if this is resolved in the latest version)\n\n- `statefulset.yaml` is highly commented to work in minikube or in prod cluster with minimal changes\n\n- Currently it works with `SecurityContext` will be changing it to work with PSP\n\n***\n#### Tip\n\n:wave: The best way to use k8s related files just search and replace `sample-sync` with the required name and you are ready to go.\n\n***\n### Minikube setup\n\n```\ndemo$ minikube service list\n|----------------------|---------------------------|--------------|--------------------------------|\n|      NAMESPACE       |           NAME            | TARGET PORT  |              URL               |\n|----------------------|---------------------------|--------------|--------------------------------|\n| default              | kubernetes                | No node port |\n| devops               | sample-sync               |              | http://192.168.64.2:32403      |\n|                      |                           |              | http://192.168.64.2:31162      |\n|                      |                           |              | http://192.168.64.2:32383      |\n|                      |                           |              | http://192.168.64.2:32412      |\n| devops               | sample-sync-headless      | No node port |\n| kube-system          | kube-dns                  | No node port |\n| kubernetes-dashboard | dashboard-metrics-scraper | No node port |\n| kubernetes-dashboard | kubernetes-dashboard      | No node port |\n|----------------------|---------------------------|--------------|--------------------------------|\n```\n\n### Enable ingress on minikube\n\n```\ndemo$ minikube addons enable ingress\nThe 'ingress' addon is enabled\n```\n\n### Headless service\n\n:wave: A Headless Service is a service when you don’t need load-balancing and a single Service IP. Instead of load-balancing it will return the IPs of the attached Pod. Headless Services do not have a Cluster IP associated. Request will not be proxied by kube-proxy, instead NiFi will handle the service discovery.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://files.gitter.im/tomarv2/KRXn/Screen-Shot-2020-04-12-at-6.33.46-PM.png\"\u003e\n\u003c/p\u003e\n\n\n#### Note:\n\n:wave: If you find any issues or wants to see anything specific please open an issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomarv2%2Fnifi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomarv2%2Fnifi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomarv2%2Fnifi/lists"}