{"id":13549216,"url":"https://github.com/chadgeary/nifi","last_synced_at":"2025-04-02T22:31:36.802Z","repository":{"id":43049559,"uuid":"280558749","full_name":"chadgeary/nifi","owner":"chadgeary","description":"Deploy a secured, clustered, auto-scaling NiFi service in AWS.","archived":true,"fork":false,"pushed_at":"2022-07-31T17:00:08.000Z","size":5171,"stargazers_count":48,"open_issues_count":2,"forks_count":14,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-03T18:34:38.579Z","etag":null,"topics":["ansible","apache","apache-nifi","automation","aws","big-data","cloud","ec2","iac","iam","kms","nifi","pipeline","rhel","s3","terraform","ubuntu","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chadgeary.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-18T01:32:11.000Z","updated_at":"2024-10-03T20:18:10.000Z","dependencies_parsed_at":"2022-07-09T06:30:19.193Z","dependency_job_id":null,"html_url":"https://github.com/chadgeary/nifi","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadgeary%2Fnifi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadgeary%2Fnifi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadgeary%2Fnifi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadgeary%2Fnifi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chadgeary","download_url":"https://codeload.github.com/chadgeary/nifi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246905111,"owners_count":20852812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","apache","apache-nifi","automation","aws","big-data","cloud","ec2","iac","iam","kms","nifi","pipeline","rhel","s3","terraform","ubuntu","zookeeper"],"created_at":"2024-08-01T12:01:19.530Z","updated_at":"2025-04-02T22:31:32.799Z","avatar_url":"https://github.com/chadgeary.png","language":"HCL","funding_links":[],"categories":["HCL"],"sub_categories":[],"readme":"# Reference\nNiFi secure+autoscaling cluster built automatically in AWS via Terraform+Ansible.\n\n# Options\nTwo designs are provided, either:\n- NiFi on EC2 with Zookeeper running within the same EC2 instances, or\n- NiFi on EC2 with Zookeeper running separately in ECS Fargate.\n- Side note - for considerations about using RHEL as opposed to Ubuntu as the base EC2 OS, see `rhel.md`.\n\n# Requirements\n- An AWS account\n- Follow Step-by-Step (compatible with Windows and Ubuntu)\n\n# Media \n- [Video Guide](https://youtu.be/7idB-OuDOd0) - a bit outdated, but still useful. Follow along with me as I deploy using the step-by-step guide below.\n- [Discord](https://discord.gg/G6W4UDJEZ3) - for questions, ideas, comments, or troubleshooting assistance.\n\n# Step-by-Step Terraform Deployment \nWindows Users install WSL (Windows Subsystem Linux)\n```\n#############################\n## Windows Subsystem Linux ##\n#############################\n# Launch an ELEVATED Powershell prompt (right click -\u003e Run as Administrator)\n\n# Enable Windows Subsystem Linux\ndism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart\n\n# Reboot your Windows PC\nshutdown /r /t 5\n\n# After reboot, launch a REGULAR Powershell prompt (left click).\n# Do NOT proceed with an ELEVATED Powershell prompt.\n\n# Download the Ubuntu 2004 package from Microsoft\ncurl.exe -L -o ubuntu-2004.appx https://aka.ms/wsl-ubuntu-2004\n \n# Rename the package\nRename-Item ubuntu-2004.appx ubuntu-2004.zip\n \n# Expand the zip\nExpand-Archive ubuntu-2004.zip ubuntu-2004\n \n# Change to the zip directory\ncd ubuntu-2004\n \n# Execute the ubuntu 2004 installer\n.\\ubuntu2004.exe\n \n# Create a username and password when prompted\n```\nInstall Terraform, Git, and create an SSH key pair\n```\n#############################\n##  Terraform + Git + SSH  ##\n#############################\n# Add terraform's apt key (enter previously created password at prompt)\ncurl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -\n \n# Add terraform's apt repository\nsudo apt-add-repository \"deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main\"\n \n# Install terraform and git\nsudo apt-get update \u0026\u0026 sudo apt-get -y install terraform git\n \n# Clone the project\ngit clone https://github.com/chadgeary/nifi\n\n# Create SSH key pair (RETURN for defaults)\nssh-keygen\n```\n\nInstall the AWS cli and create non-root AWS user. An [AWS account](https://portal.aws.amazon.com/billing/signup) is required to continue.\n```\n#############################\n##          AWS            ##\n#############################\n# Open powershell and start WSL\nwsl\n\n# Change to home directory\ncd ~\n\n# Install python3 pip\nsudo apt update \u0026\u0026 sudo DEBIAN_FRONTEND=noninteractive apt-get -q -y install python3-pip\n\n# Install awscli via pip\npip3 install --user --upgrade awscli\n\n# Create a non-root AWS user in the AWS web console with admin permissions\n# This user must be the same user running terraform apply\n# Create the user at the AWS Web Console under IAM -\u003e Users -\u003e Add user -\u003e Check programmatic access and AWS Management console -\u003e Attach existing policies -\u003e AdministratorAccess -\u003e copy Access key ID and Secret Access key\n# See for more information: https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html#getting-started_create-admin-group-console\n\n# Set admin user credentials\n~/.local/bin/aws configure\n\n# Validate configuration\n~/.local/bin/aws sts get-caller-identity \n\n# For troubleshooting EC2 instances, use the SSM Session Manager plugin\ncurl \"https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb\" -o ~/session-manager-plugin.deb\nsudo dpkg -i ~/session-manager-plugin.deb\n\n# and set the SSH helper configuration for SSM Session Manager\ntee -a ~/.ssh/config \u003c\u003c EOM\nhost i-* mi-*\n    ProxyCommand sh -c \"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'\"\nEOM\n```\n\nCustomize the deployment - See variables section below\n```\n# Change to the project's aws directory in powershell\ncd ~/nifi/zks-on-ec2/\n\n# Open File Explorer in a separate window\n# Navigate to ubuntu project directory - change \\chad\\ to your WSL username\n%HOMEPATH%\\ubuntu-2004\\rootfs\\home\\chad\\nifi\\ubuntu\n\n# Edit the nifi.tfvars file using notepad and save\n```\n\nDeploy\n```\n# In powershell's WSL window, change to the project's aws directory\ncd ~/nifi/zks-on-ec2/\n\n# Initialize terraform and apply the terraform state\nterraform init\nterraform apply -var-file=\"nifi.tfvars\"\n\n# If permissions errors appear, fix with the below command and re-run the terraform apply.\nsudo chown $USER nifi.tfvars \u0026\u0026 chmod 600 nifi.tfvars\n\n# Note the outputs from terraform after the apply completes\n\n# Wait for the virtual machine to become ready (Ansible will setup the services for us). NiFi can take 15+ minutes to initialize.\n```\n\n# Variables\n```\n# See nifi.tfvars\n```\n\n# Post-Deployment\nReview terraform output for quick links to State Manager (ansible) status, Load Balancer health, Cloudwatch logs, and the admin certificate in S3 which must be added to a browser for web access.\n\n# Maintenance\nIf modifying nifi.properties:\n1. Change the nifi.properties file in `playbooks/zookeepers/` and `playbooks/nodes/`\n2. Re-run `terraform apply -var-file=\"nifi.tfvars\"`\n3. Re-apply the SSM associations mentioned in `terraform output`\n\nIf re-sizing instances or otherwise modifying autoscaling group(s):\n1. Change the instance type in `nifi.tfvars`\n2. Re-run `terraform apply -var-file=\"nifi.tfvars\"`\n3. Scale the node autoscaling group down, either all at once (min 0 / max 0) or incrementally to replace instances of the old size/AMI.\n4. Scale the zookeeper autoscaling groups down, always leave at least one zookeeper running, preferably two - e.g.:\n  - If zk1, zk2, and zk3 are running, scale down zk3. Once complete, scale zk3 back up.\n  - Repeat for zk2, then zk3.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchadgeary%2Fnifi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchadgeary%2Fnifi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchadgeary%2Fnifi/lists"}