{"id":15374505,"url":"https://github.com/lispython/kafka-replicator","last_synced_at":"2025-09-11T17:37:40.591Z","repository":{"id":55879462,"uuid":"275628013","full_name":"Lispython/kafka-replicator","owner":"Lispython","description":"Application for replication data between kafka clusters.","archived":false,"fork":false,"pushed_at":"2020-12-09T22:47:17.000Z","size":86,"stargazers_count":4,"open_issues_count":9,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T21:42:05.089Z","etag":null,"topics":["kafka","kafka-replicator","monitoring","prometheus-exporter","prometheus-metrics","replication","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lispython.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-28T16:47:58.000Z","updated_at":"2023-03-27T07:25:18.000Z","dependencies_parsed_at":"2022-08-15T08:30:44.798Z","dependency_job_id":null,"html_url":"https://github.com/Lispython/kafka-replicator","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lispython%2Fkafka-replicator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lispython%2Fkafka-replicator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lispython%2Fkafka-replicator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lispython%2Fkafka-replicator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lispython","download_url":"https://codeload.github.com/Lispython/kafka-replicator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249094932,"owners_count":21211837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-replicator","monitoring","prometheus-exporter","prometheus-metrics","replication","rust"],"created_at":"2024-10-01T13:58:56.278Z","updated_at":"2025-04-15T15:11:49.568Z","avatar_url":"https://github.com/Lispython.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka replicator\n\n**Kafka Replicator** is an easy to use tool for copying data between two Apache Kafka clusters with configurable re-partitionning strategy.\n\nData will be read from topics in the origin cluster and written to a topic/topics in the destination cluster according config rules.\n\n\n# Features\n\nLets start with an overview of features that exist in kafka-replicator:\n\n  * [x] **Data replication:** Real-time event streaming between Kafka clusters and data centers;\n  * [ ] **Schema replication:** Copy schema from source cluster to destination;\n  * [x] **Flexible topic selection:** Select topics with configurable config;\n  * [ ] **Auto-create topics:** Destination topics are automatically created for strict_p2p strategy;\n  * [x] **Stats:** The tool shows replication status;\n  * [x] **Monitoring:** Kafka replicator exports stats via prometheus.\n  * [ ] **Cycle detection**\n\n\n\n# Use cases\n\n  * Replicate data between Kafka clusters;\n  * Aggregate record from several topics and put them into one;\n  * Extend bandwidth for exist topic via repartitioning strategy.\n\n\n# Installation\n\n\n### System dependencies\n\n``` shell\nlibsasl2-dev\nlibssl-dev\n```\n\n\n## Install from crates.io\n\n\nIf you have the Rust toolchain already installed on your local system.\n\n``` shell\nrustup update stable\ncargo install kafka-replicator\n```\n\n\n## Compile and run it from sources\n\nClone the repository and change it to your working directory.\n\n```shell\ngit clone https://github.com/lispython/kafka-replicator.git\ncd kafka-replicator\n\nrustup override set stable\nrustup update stable\ncargo install\n```\n\n\n# Usage\n\n``` shell\nRUST_LOG=info kafka-replicator /path/to/config.yml\n```\n\n## Run it using Docker\n\n\n``` shell\nsudo docker run -it -v /replication/:/replication/ -e RUST_LOG=info lispython/kafka_replicator:latest kafka-replicator /replication/config.yml\n```\n\n### Example config\n\n\n``` yaml\nclusters:\n  - name: cluster_1\n    hosts:\n      - replicator-kafka-1:9092\n      - replicator-kafka-1:9092\n  - name: cluster_2\n    hosts:\n      - replicator-kafka-2:9092\n\nclients:\n  - client: cl_1_client_1\n    cluster: cluster_1\n    config: # optional\n       message.timeout.ms: 5000\n       auto.offset.reset: earliest\n  - client: cl_2_client_1\n    cluster: cluster_2\n\nroutes:\n  - upstream_client: cl_1_client_1\n    downstream_client: cl_1_client_1\n    upstream_topics:\n      - 'topic1'\n    downstream_topic: 'topic2'\n    repartitioning_strategy: random # strict_p2p | random\n    upstream_group_id: group_22\n    show_progress_interval_secs: 10\n    limits:\n      messages_per_sec: 10000\n      number_of_messages:\n\n  - upstream_client: cl_1_client_1\n    downstream_client: cl_2_client_1\n    upstream_topics:\n      - 'topic2'\n    downstream_topic: 'topic2'\n    repartitioning_strategy: strict_p2p\n    upstream_group_id: group_22\n    show_progress_interval_secs: 10\n\n  - upstream_client: cl_2_client_1\n    downstream_client: cl_1_client_1\n    upstream_topics:\n      - 'topic2'\n    downstream_topic: 'topic3'\n    repartitioning_strategy: strict_p2p # strict_p2p | random\n    default_begin_offset: earliest # optional\n    upstream_group_id: group_2\n    show_progress_interval_secs: 10\n\n\nobservers:\n  - client: cl_1_client_1\n    name: \"my name\"\n    group_id: group_name # used for remaining metrics\n    topics: # filter by topics\n      - 'topic1'\n      - 'topic2'\n    fetch_timeout_secs: 5 # default: 5\n    fetch_interval_secs: 5 # default: 60\n    show_progress_interval_secs: 10 # default: 60\n\n  - client: cl_2_client_1\n    topic: 'topic3'\n    topics:\n      - 'topic2'\n    show_progress_interval_secs: 5\n\n\n  - client: cl_1_client_1\n    topic: 'topic1'\n    topics: [] # fetch all topics\n```\n\n\n### Options describing\n\nRoot config options:\n - _clusters_ - are a list of Kafka Clusters\n - _clients_ - are a list of configurations for consumers\n - _routes_ - are a list of replication rules\n - _observers_ - are a list of observers\n\n\n### Contributing\nAny suggestion, feedback or contributing is highly appreciated. Thank you for your support!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flispython%2Fkafka-replicator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flispython%2Fkafka-replicator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flispython%2Fkafka-replicator/lists"}