{"id":21685643,"url":"https://github.com/redis-field-engineering/redis-connect-dist","last_synced_at":"2025-08-20T20:33:02.996Z","repository":{"id":37632142,"uuid":"288628302","full_name":"redis-field-engineering/redis-connect-dist","owner":"redis-field-engineering","description":"Real-Time Event Streaming \u0026 Change Data Capture","archived":false,"fork":false,"pushed_at":"2024-10-22T18:22:17.000Z","size":41284,"stargazers_count":45,"open_issues_count":2,"forks_count":11,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-09T12:54:45.062Z","etag":null,"topics":["cdc","connect","csv","etl","etl-automation","etl-framework","etl-pipeline","event-sourcing","event-streaming","redis","rediscdc","redisconnect","replication"],"latest_commit_sha":null,"homepage":"https://redis-field-engineering.github.io/redis-connect-dist/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redis-field-engineering.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-19T03:54:55.000Z","updated_at":"2024-11-16T17:59:22.000Z","dependencies_parsed_at":"2023-12-05T18:34:54.052Z","dependency_job_id":"efe146a2-7430-4862-a851-e04b00dd63e2","html_url":"https://github.com/redis-field-engineering/redis-connect-dist","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-field-engineering%2Fredis-connect-dist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-field-engineering%2Fredis-connect-dist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-field-engineering%2Fredis-connect-dist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-field-engineering%2Fredis-connect-dist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redis-field-engineering","download_url":"https://codeload.github.com/redis-field-engineering/redis-connect-dist/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230454431,"owners_count":18228392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","connect","csv","etl","etl-automation","etl-framework","etl-pipeline","event-sourcing","event-streaming","redis","rediscdc","redisconnect","replication"],"created_at":"2024-11-25T16:22:16.386Z","updated_at":"2024-12-19T15:09:51.093Z","avatar_url":"https://github.com/redis-field-engineering.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"/images/icons/Redis Connect Banner.png\" alt=\"Redis Connect\" width = \"100%\" title=\"Redis Connect\"\u003e\u003c/p\u003e\n\n**Redis Connect** :rocket: is a distributed platform that enables real-time event streaming, transformation, and propagation of \nchanged-data events from heterogeneous data platforms to [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache/), [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/), and [Redis Enterprise](https://redis.com/redis-enterprise-software/overview/).\n\u003cbr\u003e\u003cbr\u003e :white_check_mark: No Code :white_check_mark: Seamless Integration :white_check_mark: Multi-Tenancy :white_check_mark: Linear-Scalability :white_check_mark: High-Availability :white_check_mark: Support\n\u003cbr\u003e\u003cbr\u003e\n\n## End-to-End demos\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/db2/demo/README.md\"\u003e\u003cimg src=\"/images/icons/DB2 Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect DB2 demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/mysql/demo/README.md\"\u003e\u003cimg src=\"/images/icons/MySQL Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect MySQL demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/oracle/demo/README.md\"\u003e\u003cimg src=\"/images/icons/Oracle Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Oracle demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/postgres/demo/README.md\"\u003e\u003cimg src=\"/images/icons/Postgres Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Postgres demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/mssql/demo/README.md\"\u003e\u003cimg src=\"/images/icons/SQL Server Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect SQL Server demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/mongodb/demo/README.md\"\u003e\u003cimg src=\"/images/icons/MongoDB Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect MongoDB demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/gemfire/demo/README.md\"\u003e\u003cimg src=\"/images/icons/Geode Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Gemfire/Apache Geode demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/splunk/demo/README.md\"\u003e\u003cimg src=\"/images/icons/Splunk HEC Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Splunk HEC demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/files/demo/README.md\"\u003e\u003cimg src=\"/images/icons/CSV Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Files demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"/examples/vertica/demo/README.md\"\u003e\u003cimg src=\"/images/icons/Vertica Icon.png\" style=\"float: center;\" width=\"100\" height=\"100\" alt=\"Redis Connect Vertica demo\"/\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Table of Contents\n\n* [Background](#background)\n* [Quick start](#quick-start)\n* [Requirements](#requirements)\n\n## Background\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e \u003cb\u003eCloud-Native Architecture\u003c/b\u003e \u003cbr\u003e Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.\u003c/td\u003e\n        \u003ctd width=\"50%\"\u003e\u003cimg src=\"/images/capabilities/Redis Connect Architecture.png\" style=\"float: right;\" width=\"500\" height=\"250\" title=\"Redis Connect Architecture\" alt=\"Redis Connect Architecture\"/\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\u003ctd height=\"20\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd width=\"50%\"\u003e\u003cimg src=\"/images/capabilities/Redis Insight.png\" style=\"float: right;\" width=\"500\" height=\"200\" alt=\"Redis Insight\"/\u003e\u003c/td\u003e \n        \u003ctd\u003e \u003cb\u003eMulti-Tenancy | Partitioning | Linear Scalability\u003c/b\u003e \u003cbr\u003e Redis Connect manages multi-tenant replication pipelines. A pipeline from source to sink is known as a job. Jobs with different source types can be collocated without becoming noisy neighbors. Jobs can be partitioned for linear scalability across one or more cluster nodes.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\u003ctd bgcolor=\"#FFFFFF\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e \u003cb\u003eHigh-Availability | Recovery\u003c/b\u003e \u003cbr\u003e Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the event of a node failure or network split, in-flight jobs will fail over to another node and seamlessly begin replication from the last committed checkpoint. Data is not lost, and order is preserved. Redis Connect works in container orchestration environments such as Kubernetes and OpenShift.\u003c/td\u003e\n        \u003ctd width=\"50%\"\u003e\u003cimg src=\"/images/capabilities/Redis Connect Cluster.png\" style=\"float: right;\" width=\"500\" height=\"200\" alt=\"Redis Connect Cluster\"/\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\u003ctd bgcolor=\"#FFFFFF\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd width=\"50%\"\u003e\u003cimg src=\"/images/capabilities/Redis Connect Custom Transformer.jpg\" style=\"float: none;\" width=\"500\" height=\"150\" alt=\"Redis Connect Custom Transformation\"/\u003e\u003c/td\u003e\n        \u003ctd\u003e \u003cb\u003eCustom Transformations\u003c/b\u003e \u003cbr\u003e Redis Connect jobs support user-defined business logic. You can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. You can also extend the supported list of target sinks.\u003c/td\u003e \n    \u003c/tr\u003e\n    \u003ctr\u003e\u003ctd bgcolor=\"#FFFFFF\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e \u003cb\u003eREST API | CLI | \u003ca href=\"https://redis-field-engineering.github.io/redis-connect-api-docs\" target=\"_blank\"\u003eSwagger UI\u003c/a\u003e\u003c/b\u003e \u003cbr\u003e Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. You can configure, start, stop, migrate, and restart jobs using the built-in REST API and interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.\u003c/td\u003e\n        \u003ctd width=\"50%\"\u003e\u003ca href=\"https://redis-field-engineering.github.io/redis-connect-api-docs\"\u003e\u003cimg src=\"/images/capabilities/Redis Connect Swagger UI.png\" style=\"float: right;\" width=\"500\" height=\"200\" alt=\"Redis Connect Swagger UI\"\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\u003ctd bgcolor=\"#FFFFFF\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd width=\"50%\"\u003e\u003cimg src=\"/images/capabilities/Redis Enterprise ACL.png\" style=\"float: right;\" width=\"500\" height=\"200\" alt=\"Redis Enterprise ACL\"/\u003e\u003c/td\u003e\n        \u003ctd\u003e \u003cb\u003eEnterprise-Grade Security\u003c/b\u003e \u003cbr\u003e Redis Connect jobs are stateless, so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s security, including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored in Redis Connect; these secrets can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Requirements\n\n### Minimum production hardware requirements\n\n* 1 GB of RAM\n* 4 CPU cores\n* 20 GB of disk space\n* 1 Gbps network\n\n### Runtime requirements\n\n* JRE 11+ (JRE 17+ version 0.10.7 onwards) e.g. [Azul OpenJDK](https://www.azul.com/downloads/?package=jdk#download-openjdk)\n\n## Quick Start\n\nYou can run Redis Connect as a container or by downloading the code and running in your environment of choice.\n\n### Docker\n\nYou can run and deploy Redis Connect using the [Redis Connect Docker image](https://hub.docker.com/r/redislabs/redis-connect).\n\n### Download\n\nDownload the [latest release](https://github.com/redis-field-engineering/redis-connect-dist/releases) for `Linux` or `Windows` and unarchive `redis-connect-\u003cversion\u003e.\u003cbuild\u003e.[tar.gz|zip]` archive\u003cbr\u003e\n\nThe following subdirectories will be extracted under `/redis-connect`:\n* `bin` – Startup scripts\n* `lib` – Dependencies\n* `config` – Credentials property files, jobmanager.properties, and job-config (JSON) examples\n* `extlib` – Custom/external dependencies (e.g., [custom stages](https://github.com/redis-field-engineering/redis-connect-custom-stage-demo), source-database drivers, etc.)\n\n### Launch Redis Connect\n\nRedis Connect includes scripts for launching a single instance. You can run the scripts as follows:\n\n#### On Linux\n```bash\nredis-connect/bin$ ./redisconnect.sh    \n-------------------------------\nRedis Connect startup script.\n*******************************\nPlease ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below\nCheck the value of redis.connection.url and credentials.dir.path in jobmanager.properties e.g.\nredis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836\ncredentials.dir.path=/var/secrets/redis\n*******************************\nUsage: [-h|cli|start]\noptions:\n-h: Print this help message and exit.\ncli: init Redis Connect CLI\nstart: init Redis Connect Instance (Cluster Member)\n-------------------------------\n```\n\n#### On Windows\n```cmd\nredis-connect\\bin\u003e redisconnect.bat\n```\n\n| Prerequisite Configuration :exclamation:                                                                                                                                                            |\n|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Update `credentials.dir.path` and `redis.connection.url` in `/config/jobmanager.properties`\u003cbr/\u003e Example - \u003ca href=\"/examples/postgres/demo/config/jobmanager.properties\"\u003ejobmanager.properties\u003c/a\u003e |\n\n### Starting an instance\n\nLinux:\n```bash\nredis-connect/bin$ ./redisconnect.sh start\n```\nWindows:\n```cmd\nredis-connect\\bin\u003e redisconnect.bat start\n```\n\u003cimg src=\"/images/quick-start/Redis Connect Start Log.png\" style=\"float: right;\" width = 700px height = 250px alt=\"Redis Connect Start Log\"/\u003e\n\n### Swagger UI\n\nRedis Connect Swagger UI is available on port 8282 by default. If you're running locally, you can point your browser to `http://localhost:8282/swagger-ui/index.html`\n\n\u003cbr\u003e_For quick start, use '**cdc_job**' as **jobName**_\n\u003cbr\u003e\u003cbr\u003e\u003cimg src=\"/images/quick-start/Redis Connect Swagger Front Page.jpg\" style=\"float: right;\" width = 700px height = 425px/\u003e\n\n**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`\n\u003cbr\u003e_For quick start, use a sample job configuration:_ \u003ca href=\"/examples/db2/demo/config/samples/payloads/cdc-job.json\"\u003eDB2\u003c/a\u003e, \u003ca href=\"/examples/files/demo/config/samples/payloads/cdc-job.json\"\u003eFiles\u003c/a\u003e, \u003ca href=\"/examples/gemfire/demo/config/samples/payloads/cdc-job.json\"\u003eGemfire\u003c/a\u003e, \u003ca href=\"/examples/mongodb/demo/config/samples/payloads/cdc-job.json\"\u003eMongoDB\u003c/a\u003e, \u003ca href=\"/examples/mysql/demo/config/samples/payloads/cdc-job.json\"\u003eMySQL\u003c/a\u003e, \u003ca href=\"/examples/oracle/demo/config/samples/payloads/cdc-job.json\"\u003eOracle\u003c/a\u003e, \u003ca href=\"/examples/postgres/demo/config/samples/payloads/cdc-job.json\"\u003ePostgreSQL\u003c/a\u003e, \u003ca href=\"/examples/mssql/demo/config/samples/payloads/cdc-job.json\"\u003eSQL Server\u003c/a\u003e and \u003ca href=\"/examples/vertica/demo/config/samples/payloads/cdc-job.json\"\u003eVERTICA\u003c/a\u003e\n\u003cbr\u003e\u003cbr\u003e\u003cimg src=\"/images/quick-start/Redis Connect Save Job Config.png\" style=\"float: right;\" width = 700px height = 375px/\u003e\n\n| Prerequisite Configuration :exclamation:                                                                                                                                                                                           |\n|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Update `credentialsDirectoryPath`, `databaseURL`, `database.dbname` (`database.names` for SQL Server), `database.hostname`, `database.port` and `columns` within sample job configuration for source and target, where applicable. |\n\n**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`\n\u003cbr\u003e_For quick start, use '**stream**' as **jobType**_\n\u003cbr\u003e\u003cbr\u003e\u003cimg src=\"/images/quick-start/Redis Connect Start Job.png\" style=\"float: right;\" width = 700px height = 375px/\u003e\n\n**Confirm Job Claim -** `/connect/api/vi/jobs/claim/{jobStatus}`\n\u003cbr\u003e_For quick start, use '**all**' as **jobStatus**_\n\u003cbr\u003e\u003cbr\u003e\u003cimg src=\"/images/quick-start/Redis Connect Get Claims.png\" style=\"float: right;\" width = 700px height = 250px/\u003e\n\nOnce you've configured a job, try inserting some records into the source database. Then confirm that they have arrived in Redis.\n\n## Copyright\n\nRedis Connect is developed by Redis, Inc. Copyright (C) 2023 Redis, Inc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis-field-engineering%2Fredis-connect-dist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredis-field-engineering%2Fredis-connect-dist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis-field-engineering%2Fredis-connect-dist/lists"}