{"id":13646796,"url":"https://github.com/salesforce/mirus","last_synced_at":"2025-04-12T19:42:34.612Z","repository":{"id":33267028,"uuid":"140903124","full_name":"salesforce/mirus","owner":"salesforce","description":"Mirus is a cross data-center data replication tool for Apache Kafka","archived":false,"fork":false,"pushed_at":"2025-03-18T20:30:23.000Z","size":275,"stargazers_count":203,"open_issues_count":32,"forks_count":43,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-04-03T22:09:29.486Z","etag":null,"topics":["kafka","kafka-connect","mirroring"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-14T00:14:10.000Z","updated_at":"2025-01-03T21:47:18.000Z","dependencies_parsed_at":"2023-09-24T16:57:20.453Z","dependency_job_id":"30a3eb1b-1e8a-430e-895e-4106f777302d","html_url":"https://github.com/salesforce/mirus","commit_stats":{"total_commits":114,"total_committers":15,"mean_commits":7.6,"dds":0.5877192982456141,"last_synced_commit":"5d68c778582c6d7c18b0198b2fe37818347746f5"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fmirus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fmirus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fmirus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fmirus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce","download_url":"https://codeload.github.com/salesforce/mirus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625325,"owners_count":21135511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-connect","mirroring"],"created_at":"2024-08-02T01:03:06.520Z","updated_at":"2025-04-12T19:42:34.579Z","avatar_url":"https://github.com/salesforce.png","language":"Java","readme":"[![Build Status](https://travis-ci.org/salesforce/mirus.svg?branch=master)](https://travis-ci.org/salesforce/mirus)\n[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\n# Mirus\n\nA tool for distributed, high-volume replication between [Apache Kafka](https://kafka.apache.org/documentation) clusters based\non [Kafka Connect](https://kafka.apache.org/documentation/#connect). Designed for easy\noperation in a high-throughput, multi-cluster environment.\n\n## Features\n\n- **Dynamic Configuration:** Uses the Kafka Connect REST API for dynamic API-driven configuration\n- **Precise Replication:** Supports a regex whitelist and an explicit topic whitelist\n- **Simple Management for Multiple Kafka Clusters:** Supports multiple source clusters with one Worker process\n- **Continuous Ingestion:** Continues consuming from the source cluster while flushing and committing offsets\n- **Built for Dynamic Kafka Clusters:** Able to handle topics and partitions being created and deleted in source and destination clusters\n- **Scalable:** Creates a configurable set of worker tasks that are distributed across a Kafka\nConnect cluster for high performance, even when pushing data over the internet\n- **Fault tolerant:** Includes a monitor thread that looks for task failures and optionally auto-restarts\n- **Monitoring:** Includes custom JMX metrics for production ready monitoring and alerting\n\n## Overview\n\nMirus is built around Apache Kafka Connect, providing `SourceConnector` and `SourceTask` implementations\noptimized for reading data from Kafka source clusters. The `MirusSourceConnector` runs a\n`KafkaMonitor` thread,  which  monitors the source and destination Apache Kafka cluster\npartition allocations, looking for changes and applying a configurable topic whitelist.  Each task\nis responsible for a subset of the matching partitions, and runs an independent `KafkaConsumer` and\n`KafkaProducer` client pair to do the work of replicating those partitions.\n\nTasks can be restarted independently without otherwise affecting a running cluster, are monitored\ncontinuously for failure, and are optionally automatically restarted.\n\nTo understand how Mirus distributes work across a cluser of machines please read the [Kafka Connect documentation](https://kafka.apache.org/documentation/#connect).\n\n\n## Installation\n\nTo build a package containing the Mirus jar file and all dependencies, run `mvn package -P all`:\n\n- `target/mirus-${project.version}-all.zip`\n\nThis package can be unzipped for use (see [Quick Start](#quick-start)).\n\nMaven also builds the following artifacts when you run `mvn package`. These are useful if you need\ncustomized packaging for your own environment:\n\n- `target/mirus-${project.version}.jar`: Primary Mirus jar (dependencies not included)\n- `target/mirus-${project.version}-run.zip`: A package containing the Mirus run control scripts\n\n\n## Usage\n\nThese instructions assume you have expanded the `mirus-${project.version}-all.zip` package.\n\n### Mirus Worker Instance\n\nA single Mirus Worker can be started using this helper script.\n\n```\n\u003e bin/mirus-worker-start.sh [worker-properties-file]\n```\n\n`worker-properties-file`: Path to the Mirus `worker properties file`, which configures the Kafka Connect framework. See [quickstart-worker.properties](config/quickstart-worker.properties) for an example.\n\n#### Options:\n `--override property=value`: optional command-line override for any item in the Mirus worker properties file.  Multiple override options are supported (similar to the equivalent flag in Kafka).\n\n### Mirus Offset Tool\n\nMirus includes a simple tool for reading and writing offsets. This can be useful for migration\nfrom other replication tools, for debugging, and for offset monitoring in production. The tool supports\nCSV and JSON input and output.\n\nFor detailed usage:\n\n```\n\u003e bin/mirus-offset-tool.sh --help\n```\n\n\n## Quick Start\n\nTo run the Quick Start example you will need running Kafka and Zookeeper clusters to work with.  We will assume\nyou have a standard Apache Kafka Quickstart test cluster running on localhost. Follow the [Kafka Quick Start instructions](https://kafka.apache.org/quickstart).\n\nFor this tutorial we will set up a Mirus worker instance to mirror the topic `test` in loop-back mode to\nanother topic in the same cluster. To avoid a conflict the destination topic name will be set to\n`test.mirror` using the `destination.topic.name.suffix` configuration option.\n\n\n1. Build the full Mirus project using Maven\n\n    ```\n    \u003e mvn package -P all\n    ```\n\n1. Unpack the Mirus \"all\" package\n\n    ```\n    \u003e mkdir quickstart; cd quickstart; unzip ../target/mirus-*-all.zip\n    ```\n\n1. Start the `quickstart` worker using the sample worker properties file\n\n    ```\n    \u003e bin/mirus-worker-start.sh config/quickstart-worker.properties\n\n    ```\n\n1. In another terminal, confirm the Mirus Kafka Connect REST API is running\n    ```\n    \u003e curl localhost:8083\n\n    {\"version\":\"1.1.0\",\"commit\":\"fdcf75ea326b8e07\",\"kafka_cluster_id\":\"xdxNfx84TU-ennOs7EznZQ\"}\n    ```\n\n1. Submit a new `MirusSourceConnector` configuration to the REST API with the name `mirus-quickstart-source`\n\n    ```\n    \u003e curl localhost:8083/connectors/mirus-quickstart-source/config \\\n          -X PUT \\\n          -H 'Content-Type: application/json' \\\n          -d '{\n               \"name\": \"mirus-quickstart-source\",\n               \"connector.class\": \"com.salesforce.mirus.MirusSourceConnector\",\n               \"tasks.max\": \"5\",\n               \"topics.whitelist\": \"test\",\n               \"destination.topic.name.suffix\": \".mirror\",\n               \"destination.consumer.bootstrap.servers\": \"localhost:9092\",\n               \"consumer.bootstrap.servers\": \"localhost:9092\",\n               \"consumer.client.id\": \"mirus-quickstart\",\n               \"consumer.key.deserializer\": \"org.apache.kafka.common.serialization.ByteArrayDeserializer\",\n               \"consumer.value.deserializer\": \"org.apache.kafka.common.serialization.ByteArrayDeserializer\"\n           }'\n    ```\n1. Confirm the new connector is running\n    ```\n    \u003e curl localhost:8083/connectors\n\n    [\"mirus-quickstart-source\"]\n    ```\n    ```$bash\n    \u003e curl localhost:8083/connectors/mirus-quickstart-source/status\n\n    {\"name\":\"mirus-quickstart-source\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"1.2.3.4:8083\"},\"tasks\":[],\"type\":\"source\"}\n    ```\n\n1. Create source and destination topics `test` and `test.mirror` in your Kafka cluster\n    ```\n    \u003e cd ${KAFKA_HOME}\n    \n    \u003e bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic 'test' --partitions 1 --replication-factor 1\n    Created topic \"test\".\n    \n    \u003e bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic 'test.mirror' --partitions 1 --replication-factor 1\n    Created topic \"test.mirror\".\n    ```\n\n1. Mirus should detect that the new source and destination topics are available and create a new Mirus Source Task:\n\n    ```\n    \u003e curl localhost:8083/connectors/mirus-quickstart-source/status\n\n    {\"name\":\"mirus-quickstart-source\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"10.126.22.44:8083\"},\"tasks\":[{\"state\":\"RUNNING\",\"id\":0,\"worker_id\":\"10.126.22.44:8083\"}],\"type\":\"source\"}\n    ```\n\nAny message you write to the topic `test` will now be mirrored to `test.mirror`.\n\n\n## REST API\n\nSee the documentation for [Kafka Connect REST API](https://kafka.apache.org/documentation/#connect_rest).\n\n## Configuration\n\n## Kafka Connect Configuration\n\nMirus shares most Worker and Source configuration with the Kafka Connect framework. For general\ninformation on configuring the framework see:\n\n- [Kafka Connect Configuration](https://kafka.apache.org/documentation/#connect_configuring)\n- [Kafka Connect Worker Configuration](https://kafka.apache.org/documentation/#connectconfigs)\n\n\n## Mirus Specific Configuration\n\nMirus-specific configuration properties are documented in these files:\n\n- [Mirus Source Properties](src/main/java/com/salesforce/mirus/config/SourceConfigDefinition.java)\nThese can be added to the JSON config object posted to the REST API `/config` endpoint to configure a\nnew MirusSourceConnector instance. In addition, the Kafka Consumer instances\ncreated by Mirus Tasks can be configured using a `consumer.` prefix on the standard\n[Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs). The equivalent\nKafkaProducer options are configured in the Mirus Worker Properties file (see below). The\n  `destination.consumer.`prefix can be used to override the properties of the KafkaConsumer that connects to\n   the destination Kafka cluster.\n\n- [Mirus Worker Properties](src/main/java/com/salesforce/mirus/config/MirusConfigDefinition.java)\nThese are Mirus extensions to the Kafka Connect configuration, and should be applied to the\nWorker Properties file provided at startup. The Kafka Producer instances created by Kafka Connect\ncan also be configured using a `producer.` prefix on the standard\n[Kafka Producer properties](https://kafka.apache.org/documentation/#producerconfigs).\n\n## Destination Topic Checking\n\nBy default, Mirus checks that the destination topic exists in the destination Kafka cluster before\nstarting to replicate data to it. This feature can be disabled by setting the\n[enable.destination.topic.checking](src/main/java/com/salesforce/mirus/config/SourceConfigDefinition.java#L66)\nconfig option to `false`.\n\nAs of version 0.2.0, destination topic checking can also support topic re-routing performed by the\n[RegexRouter](https://kafka.apache.org/documentation/#connect_transforms) Single-Message Transformation.\nNo other `Router` Transformations are supported, so destination topic checking must be disabled in order\nto use them.\n\n## Metrics\n\nMirus produces some custom metrics in addition to the standard Kafka Connect metrics.\n\nJMX Queries are as follows\n\n### Latency (MirrorJmxReporter)\n\n```\nobjectName=\"mirus:type=MirusSourceConnector,topic=*\" attribute=\"replication-latency-ms-max\"\nobjectName=\"mirus:type=MirusSourceConnector,topic=*\" attribute=\"replication-latency-ms-min\"\nobjectName=\"mirus:type=MirusSourceConnector,topic=*\" attribute=\"replication-latency-ms-avg\"\nobjectName=\"mirus:type=MirusSourceConnector,topic=*\" attribute=\"replication-latency-ms-count\"\n```\n\n### Destination Information (MissingPartitionsJmxReporter)\n\n```\nobjectName=\"mirus:type=mirus\" attribute=\"missing-dest-partitions-count\"\n```\n\n### Connector Metrics (ConnectorJmxReporter)\n\n```\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"task-failed-restart-attempts-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"connector-failed-restart-attempts-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"failed-task-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"paused-task-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"destroyed-task-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"running-task-count\"\nobjectName=\"mirus:type=connector-metrics,connector=*\" attribute=\"unassigned-task-count\"\n```\n\n### Task Metrics (TaskJmxReporter)\n\n```\nobjectName=\"mirus:type=connector-task-metrics,connector=*\" attribute=\"task-failed-restart-attempts-count\"\n```\n\n## Developer Info\n\nTo preform a release: `mvn release:prepare release:perform`\nGPG Keys may need to be passed to maven with `-Darguments='-Dgpg.passphrase= -Dgpg.keyname=55Z32RD1'`\n\n# Discussion\n\nQuestions or comments can also be posted on the Mirus Github issues page.\n\n# Maintainers\n\n[Paul Davidson](http://github.com/pdavidson100) and [contributors](https://github.com/pdavidson/mirus/graphs/contributors).\n\n# Code Style\nThis project uses the [Google Java Format](https://github.com/google/google-java-format).\n","funding_links":[],"categories":["Libraries","Java","进程间通信","Operations"],"sub_categories":["Kafka","Mirroring"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fmirus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce%2Fmirus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fmirus/lists"}