Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/devopscorner/nifi
Production Grade Nifi & Nifi Registry. Deploy for VM (Virtual Machine) with Terraform + Ansible, Helm & Helmfile for Kubernetes (EKS)
https://github.com/devopscorner/nifi
ansible data-science data-structures docker docker-compose dockerhub ecr eks eks-cluster etl kubernetes machine-learning ml mlops nifi nifi-registry terraform vpn vpn-client
Last synced: 2 months ago
JSON representation
Production Grade Nifi & Nifi Registry. Deploy for VM (Virtual Machine) with Terraform + Ansible, Helm & Helmfile for Kubernetes (EKS)
- Host: GitHub
- URL: https://github.com/devopscorner/nifi
- Owner: devopscorner
- License: apache-2.0
- Created: 2022-11-20T10:33:31.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2024-10-29T12:46:58.000Z (3 months ago)
- Last Synced: 2024-10-29T15:15:22.332Z (3 months ago)
- Topics: ansible, data-science, data-structures, docker, docker-compose, dockerhub, ecr, eks, eks-cluster, etl, kubernetes, machine-learning, ml, mlops, nifi, nifi-registry, terraform, vpn, vpn-client
- Language: Python
- Homepage: https://devopscorner.id
- Size: 12.4 MB
- Stars: 9
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: Changelog.md
- License: LICENSE
Awesome Lists containing this project
README
# DevOpsCorner Nifi
[](https://img.shields.io/github/license/devopscorner/nifi)
Container Nifi & Nifi Registry
- Provisioning with Terraform & Ansible
- EC2 Docker Compose & Kubernetes (EKS)
- Python3 & Libraries
- Unified Machine Learning (ML) Framework with [Ivy](https://github.com/unifyai/ivy)
---
## Introduction to NIFI
### NIFI Usecase
- Apache NiFi Tutorial: What is NiFi? Architecture & Installation (guru99.com), go to [this](https://www.guru99.com/apache-nifi-tutorial.html) link
- An Introduction to Apache NiFi, Use Cases and Best Practices (capgemini.github.io), go to [this](https://capgemini.github.io/development/introduction-nifi-best-practices) link
- Apache NiFi - Designing a flow for a real use case - Digital Hub, go to [this](https://scc-digitalhub.github.io/platform/nifi-tutorial-gtfs) link
## Available Tags
| Containers | Pulls |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Container `devopscorner/nifi` [Tags](docs/README-Nifi.md) | [](https://hub.docker.com/r/devopscorner/nifi/) |
| Container `devopscorner/nifi-registry` [Tags](docs/README-Nifi-Registry.md) | [](https://hub.docker.com/r/devopscorner/nifi-registry/) |
## Features
- Nifi (latest: 1.18.0)
- Nifi Registry (latest: 1.18.0)
- Python-3.10.8
- Python Libraries
- cffi
- awscli
- PyYaml
- Jinja2
- httplib2
- six
- requests
- boto3
- pandas
- beautifulsoup4
- lxml
- scrapy
- SQLAlchemy
- psycopg2-binary
- ivy-core
- checkov
- Unix Tools
- git
- curl
- wget
- vim
- nano
- zip
- unzip
- ping
- netstat
---
## Prerequirements
- Ansible (`ansible`)
- Docker (`docker`)
- Docker Compose (`docker-compose`)
- AWS Cli version 2 (`aws`)
- Terraform Cli (`terraform`)
- Terraform Environment (`tfenv`)
## Documentation
- Index Documentation, go to [this](docs/README.md) link
- Ansible Documentation, go to [this](docs/README-Ansible.md) link
- Deploy Ansible Nifi, go to [this](docs/Deploy-Ansible.md) link
- Docker-Compose Nifi, go to [this](docs/Docker-Compose-Nifi.md) link
- EKS Nifi, go to [this](docs/EKS-Nifi.md) link
## Ansible Features
Ansible Services Pattern:
- Ansible Static & Dynamic Inventory
- Playbooks Group by Services
- Datadog
- Docker
- Java
- Nifi
- Roles by Ansible Galaxy
- Adminer
- Amazon-AWS
- AWS-CLI
- CertBot
- Datadog
- Docker
- Java
- Kubectl
## Terraform Features
Multi Environment Workspace:
- Remote State Terraform (S3 & DynamoDB)
- Core Infrastructure
- VPC
- Subnet EC2 & EKS
- Security Group
- NAT Gateway
- Internet Gateway
- VPC Peers Single CIDR
- VPC Peers Multi CIDR
- Resources Other Infra
- Budget
- AWS Elastic Computing (EC2)
- Jumphost
- PostgreSQL (PSQL)
- Nifi
- Amazon Relational Database Service (RDS)
## Tested Environment
### Versioning
- Docker version
```
docker -v
---
Docker version 20.10.17-rd, build c2e4e01
docker version
---
Client:
Version: 20.10.17-rd
API version: 1.41
Go version: go1.17.11
Git commit: c2e4e01
Built: Fri Jul 22 18:31:17 2022
OS/Arch: darwin/amd64
Context: default
Experimental: true
Server: Docker Desktop 4.14.1 (91661)
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go9.7
Git commit: 3056208
Built: Tue Oct 25 18:00:19 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.9
GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
- Docker-Compose version
```
docker-compose -v
---
Docker Compose version v2.11.1
```
- AWS Cli
```
aws --version
---
nifi/2.8.7 Python/3.9.11 Darwin/21.6.0 exe/x86_64 prompt/off
```
- Terraform Cli
```
terraform version
---
Terraform v1.3.5
on darwin_amd64
- provider registry.terraform.io/hashicorp/aws v3.74.3
- provider registry.terraform.io/hashicorp/local v2.1.0
- provider registry.terraform.io/hashicorp/null v3.1.0
- provider registry.terraform.io/hashicorp/random v3.1.0
- provider registry.terraform.io/hashicorp/time v0.7.2
```
- Terraform Environment Cli
```
tfenv -v
---
tfenv 2.2.2
```
## Security Check
Make sure that you didn't push sensitive information in this repository
- [ ] AWS Credentials (AWS_ACCESS_KEY, AWS_SECRET_KEY)
- [ ] AWS Account ID
- [ ] AWS Resources ARN
- [ ] Username & Password
- [ ] Private (id_rsa) & Public Key (id_rsa.pub)
- [ ] DNS Zone ID
- [ ] APP & API Key
## Copyright
- Author: **Dwi Fahni Denni (@zeroc0d3)**
- Vendor: **DevOps Corner Indonesia (devopscorner.id)**
- License: **Apache v2**