https://github.com/george0st/Ff2Cql
A simple transfer data between NiFi and CQL solution (support Apache Cassandra, ScyllaDB, AstraDB, etc.).
https://github.com/george0st/Ff2Cql
apache-cassandra apache-nifi astradb cassandra cql etl nifi nifi-processor orchestrator scylladb
Last synced: about 1 month ago
JSON representation
A simple transfer data between NiFi and CQL solution (support Apache Cassandra, ScyllaDB, AstraDB, etc.).
- Host: GitHub
- URL: https://github.com/george0st/Ff2Cql
- Owner: george0st
- License: apache-2.0
- Created: 2024-12-19T09:50:28.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-08-05T17:31:32.000Z (2 months ago)
- Last Synced: 2025-08-05T19:25:59.331Z (2 months ago)
- Topics: apache-cassandra, apache-nifi, astradb, cassandra, cql, etl, nifi, nifi-processor, orchestrator, scylladb
- Language: Java
- Homepage:
- Size: 80.7 MB
- Stars: 65
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Ff2Cql
A simple transfer data between NiFi and CQL (support Apache Cassandra,
ScyllaDB, AstraDB, YugabyteDB, etc.). The implementation details:
- development NiFi v2 processor (with controller) and java application (support Java 17/21+)
- support Apache Cassandra v4/v5, ScyllaDB, AstraDB, YugabyteDB based on CQL (Cassandra Query Language)
## 1. The main motivation
- the Apache NiFi v2 does not support Apache Cassandra v4/v5 (NiFi v2 removed
the Cassandra processor due to security vulnerabilities and unmaintained
code. The processor supported only Cassandra v3 not newer.)
## 2. Usage in NiFi
You can use this preferred way (NiFi v2 processor with controller):
- ✅ [**PutCQL**](./nifi/cql-processor/docs/README.md#3-putcql-nifi-processor), put data to CQL solution, where inputs are FlowFiles
- ✅ [**GetCQL**](./nifi/cql-processor/docs/README.md#4-getcql-nifi-processor), get data from CQL solution, where outputs are FlowFiles
- NOTE:
- You can download the [latest version](./nifi/cql-processor/output/), see 'nifi-cql-nar-*.nar' file
- Sample Flow definitions, [see](./nifi/cql-processor/docs/flow.md)
or two other older alternative ways:
- ✅ [**ExecuteProcess**](./console_app/Ff2Cql/docs/README.md#2-executeprocess-java-application) with java application (see 'Ff2Cql-*.jar'), where inputs are CSV files
- ✅ [**ExecuteStreamCommand**](./console_app/Ff2Cql/docs/README.md#3-executestreamcommand-java-application) with java application (see 'Ff2Cql-*.jar'), where inputs are FlowFiles via stdin
## 3. Connection setting
You can see a sample of relevant setting for CQL controller here
(these connection settings are used directly in unit tests):
### 3.1 Cassandra
- [Cassandra setting](./nifi/cql-processor/nifi-cql/src/test/test-cassandra.json)
- access with IP addresses (name/password)
### 3.2 Scylla
- [Scylla setting](./nifi/cql-processor/nifi-cql/src/test/test-scylla.json)
- access with IP address (without name/password)
### 3.3 AstraDB
- [AstraDB setting](./nifi/cql-processor/nifi-cql/src/test/test-astra.json)
- access with Security Connection Bundle (SCB) and Token
- NOTE about SCB: see the file with **'.zip'** suffix
- How to get the SCB? Go to the https://astra.datastax.com/...,
Menu Databases/Your database/Region/[Download SCB](./docs/assets/astradb-download-SCB.png)
- NOTE about token: see the file with **'token.json'** suffix
- How to get the token? Go to the https://astra.datastax.com/...,
Menu Tokens/Generate Token/Token Details/[Download Token Details](./docs/assets/astradb-download-token.png)
### 3.4 YugabyteDB
- [YugabyteDB setting](./nifi/cql-processor/nifi-cql/src/test/test-yugabyte.json)
- access with IP address (without name/password)
## 4. Apache NiFi v2 & Scylla/Cassandra v5
[](https://fivekg.onrender.com/images/ApacheNiFiv2&Scylla-Cassandra.mp4)