Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/redis-field-engineering/redis-connect-dist
Real-Time Event Streaming & Change Data Capture
https://github.com/redis-field-engineering/redis-connect-dist
cdc connect csv etl etl-automation etl-framework etl-pipeline event-sourcing event-streaming redis rediscdc redisconnect replication
Last synced: about 1 month ago
JSON representation
Real-Time Event Streaming & Change Data Capture
- Host: GitHub
- URL: https://github.com/redis-field-engineering/redis-connect-dist
- Owner: redis-field-engineering
- License: other
- Created: 2020-08-19T03:54:55.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-22T18:22:17.000Z (3 months ago)
- Last Synced: 2024-12-09T12:54:45.062Z (about 1 month ago)
- Topics: cdc, connect, csv, etl, etl-automation, etl-framework, etl-pipeline, event-sourcing, event-streaming, redis, rediscdc, redisconnect, replication
- Language: Shell
- Homepage: https://redis-field-engineering.github.io/redis-connect-dist/
- Size: 39.4 MB
- Stars: 45
- Watchers: 6
- Forks: 11
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
**Redis Connect** :rocket: is a distributed platform that enables real-time event streaming, transformation, and propagation of
changed-data events from heterogeneous data platforms to [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache/), [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/), and [Redis Enterprise](https://redis.com/redis-enterprise-software/overview/).
:white_check_mark: No Code :white_check_mark: Seamless Integration :white_check_mark: Multi-Tenancy :white_check_mark: Linear-Scalability :white_check_mark: High-Availability :white_check_mark: Support
## End-to-End demos
## Table of Contents
* [Background](#background)
* [Quick start](#quick-start)
* [Requirements](#requirements)
## Background
Cloud-Native Architecture
Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.
Multi-Tenancy | Partitioning | Linear Scalability
Redis Connect manages multi-tenant replication pipelines. A pipeline from source to sink is known as a job. Jobs with different source types can be collocated without becoming noisy neighbors. Jobs can be partitioned for linear scalability across one or more cluster nodes.
High-Availability | Recovery
Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the event of a node failure or network split, in-flight jobs will fail over to another node and seamlessly begin replication from the last committed checkpoint. Data is not lost, and order is preserved. Redis Connect works in container orchestration environments such as Kubernetes and OpenShift.
Custom Transformations
Redis Connect jobs support user-defined business logic. You can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. You can also extend the supported list of target sinks.
REST API | CLI | Swagger UI
Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. You can configure, start, stop, migrate, and restart jobs using the built-in REST API and interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.
Enterprise-Grade Security
Redis Connect jobs are stateless, so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s security, including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored in Redis Connect; these secrets can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.
## Requirements
### Minimum production hardware requirements
* 1 GB of RAM
* 4 CPU cores
* 20 GB of disk space
* 1 Gbps network
### Runtime requirements
* JRE 11+ (JRE 17+ version 0.10.7 onwards) e.g. [Azul OpenJDK](https://www.azul.com/downloads/?package=jdk#download-openjdk)
## Quick Start
You can run Redis Connect as a container or by downloading the code and running in your environment of choice.
### Docker
You can run and deploy Redis Connect using the [Redis Connect Docker image](https://hub.docker.com/r/redislabs/redis-connect).
### Download
Download the [latest release](https://github.com/redis-field-engineering/redis-connect-dist/releases) for `Linux` or `Windows` and unarchive `redis-connect-..[tar.gz|zip]` archive
The following subdirectories will be extracted under `/redis-connect`:
* `bin` – Startup scripts
* `lib` – Dependencies
* `config` – Credentials property files, jobmanager.properties, and job-config (JSON) examples
* `extlib` – Custom/external dependencies (e.g., [custom stages](https://github.com/redis-field-engineering/redis-connect-custom-stage-demo), source-database drivers, etc.)
### Launch Redis Connect
Redis Connect includes scripts for launching a single instance. You can run the scripts as follows:
#### On Linux
```bash
redis-connect/bin$ ./redisconnect.sh
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below
Check the value of redis.connection.url and credentials.dir.path in jobmanager.properties e.g.
redis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836
credentials.dir.path=/var/secrets/redis
*******************************
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------
```
#### On Windows
```cmd
redis-connect\bin> redisconnect.bat
```
| Prerequisite Configuration :exclamation: |
|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Update `credentials.dir.path` and `redis.connection.url` in `/config/jobmanager.properties`
Example - jobmanager.properties |
### Starting an instance
Linux:
```bash
redis-connect/bin$ ./redisconnect.sh start
```
Windows:
```cmd
redis-connect\bin> redisconnect.bat start
```
### Swagger UI
Redis Connect Swagger UI is available on port 8282 by default. If you're running locally, you can point your browser to `http://localhost:8282/swagger-ui/index.html`
_For quick start, use '**cdc_job**' as **jobName**_
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
_For quick start, use a sample job configuration:_ DB2, Files, Gemfire, MongoDB, MySQL, Oracle, PostgreSQL, SQL Server and VERTICA
| Prerequisite Configuration :exclamation: |
|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Update `credentialsDirectoryPath`, `databaseURL`, `database.dbname` (`database.names` for SQL Server), `database.hostname`, `database.port` and `columns` within sample job configuration for source and target, where applicable. |
**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`
_For quick start, use '**stream**' as **jobType**_
**Confirm Job Claim -** `/connect/api/vi/jobs/claim/{jobStatus}`
_For quick start, use '**all**' as **jobStatus**_
Once you've configured a job, try inserting some records into the source database. Then confirm that they have arrived in Redis.
## Copyright
Redis Connect is developed by Redis, Inc. Copyright (C) 2023 Redis, Inc.