{"id":51108401,"url":"https://github.com/pneff93/replicator-aggregation","last_synced_at":"2026-06-24T15:32:02.969Z","repository":{"id":210421752,"uuid":"724530385","full_name":"pneff93/replicator-aggregation","owner":"pneff93","description":"Repository to aggregate data from multiple local Kafka clusters into central CC one using Replicator","archived":false,"fork":false,"pushed_at":"2023-12-29T10:42:04.000Z","size":191,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-12-29T11:33:42.655Z","etag":null,"topics":["confluent-cloud","kafka","replicator"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pneff93.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-28T09:14:23.000Z","updated_at":"2023-12-02T15:32:26.000Z","dependencies_parsed_at":"2023-12-02T17:39:22.586Z","dependency_job_id":null,"html_url":"https://github.com/pneff93/replicator-aggregation","commit_stats":null,"previous_names":["pneff93/replicator-aggregation"],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/pneff93/replicator-aggregation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pneff93%2Freplicator-aggregation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pneff93%2Freplicator-aggregation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pneff93%2Freplicator-aggregation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pneff93%2Freplicator-aggregation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pneff93","download_url":"https://codeload.github.com/pneff93/replicator-aggregation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pneff93%2Freplicator-aggregation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34739426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-cloud","kafka","replicator"],"created_at":"2026-06-24T15:32:02.905Z","updated_at":"2026-06-24T15:32:02.963Z","avatar_url":"https://github.com/pneff93.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Aggregation via Replicator\n\nWe would like to run [Replicator](https://docs.confluent.io/platform/current/multi-dc-deployments/replicator/index.html) on several local Kafka clusters and replicate the data\ninto a central cluster on CC.\nThis repository serves as an example for **one** local cluster aggregation. \nHowever, this configuration was successfully tested for multiple local\nKafka clusters.\n\nAlso, we will run Replicator as an executable and data is emitted \nwith an Avro format.\n\n![](./images/dataAggregation.png)\n\n## Create Topic \u0026 Register Schema in CC\n\nWe first create the Kafka topic and register the schema in CC.\n\n:warning: **Important:** The schema in CC and on all local Kafka clusters are decoupled,\nmeaning that schema evolution needs to be adjusted on all cluster manually.\nWe could think of using [Schema Linking](https://docs.confluent.io/platform/current/schema-registry/schema-linking-cp.html) to \nmigrate schemas, but it would require to set up one\nSchema Exporter per local Kafka cluster in CC because CC is assumed as the single\nsource of truth.\n\n![](./images/schemaCreation.png)\n\n## Start the environment\nWe start a local Kafka cluster, a Schema Registry and Replicator as an executable\n```\ndocker-compose up -d\n```\n\n## Produce Data locally\nWe produce data to the local Kafka cluster with an Avro schema.\nThe schema is created automatically on the local SR.\n\n```\n./gradlew run\n```\n\nWe verify it is produced with a schema on the local Cluster\n```shell\nkafka-avro-console-consumer --bootstrap-server localhost:9092 --topic sensor-data-raw --property schema.registry.url=\"http://localhost:8081\"\n```\n\n## Consume data in CC\n\nWe verify it is migrated with a schema also on CC\n\n```shell\nkafka-avro-console-consumer --bootstrap-server \u003cCC Bootstrap Server\u003e --consumer.config client.properties --topic sensor-data-raw --property schema.registry.url=\"\u003cCC SR\u003e\" --property schema.registry.basic.auth.user.info=\"\u003cAPI KEY: API SECRET\u003e\" --property basic.auth.credentials.source=\"USER_INFO\"\n```\n\n\n## Important Notes\n\n* Only configure the `--cluster.id` for each Replicator, no `client.id` or `group.id` when running Replicator as an executable.\n* The internal topics (config, offset, status) will be created on the destination cluster. If you want to have the internal topics created on the source cluster we need to create the Replicator\n  as a Connector.\n* Do not sync the topic configuration.\n* We connect the `src.value.converter` with the local SR, whereas the `value.converter` is connected to the CC SR.\n* The used SA has full permissions, to restrict it following the least privilege guidance, see the corresponding [ACLs documentation](https://docs.confluent.io/platform/current/multi-dc-deployments/replicator/index.html#acls-to-write-to-the-destination-cluster)\n\nFor more information check out [replicator.properties](./Replicator/replicator.properties).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpneff93%2Freplicator-aggregation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpneff93%2Freplicator-aggregation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpneff93%2Freplicator-aggregation/lists"}