{"id":23086809,"url":"https://github.com/papirosko/mm2-lag","last_synced_at":"2025-07-19T19:40:51.077Z","repository":{"id":182748889,"uuid":"669029604","full_name":"papirosko/mm2-lag","owner":"papirosko","description":"Measure the lag for the topics, synced with MirrorMaker2","archived":false,"fork":false,"pushed_at":"2024-04-24T05:54:31.000Z","size":47,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T04:44:54.054Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/papirosko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-21T07:15:03.000Z","updated_at":"2024-04-24T05:54:35.000Z","dependencies_parsed_at":"2024-04-24T07:00:01.418Z","dependency_job_id":null,"html_url":"https://github.com/papirosko/mm2-lag","commit_stats":null,"previous_names":["papirosko/mm2-lag"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fmm2-lag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fmm2-lag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fmm2-lag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fmm2-lag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/papirosko","download_url":"https://codeload.github.com/papirosko/mm2-lag/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247033815,"owners_count":20872532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T19:31:14.462Z","updated_at":"2025-04-03T16:19:46.164Z","avatar_url":"https://github.com/papirosko.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Based on [mm2-lag-exporter](https://github.com/lowes/mm2-lag-exporter).\n\n# Mirrormaker 2\nApache Kafka MirrorMaker2 replicates data across two Kafka clusters, within or across data centers. \nMirrorMaker takes messages from a source Kafka cluster and writes them to a target Kafka cluster.\n\n# MM2 Lag Exporter\nThis application monitors the mirror maker 2 replication process and calculates the mirroring lag between \nsource and target cluster topics.These lag details are exposed as REST endpoints and as well as prometheus \nmetrics which can be visualized in grafana dashboard.\n\n# Clusters configuration\nAll configuration should be stored in a file `clusters.yaml`\n\nCheck `cluster-demo.yaml`. It contains 2 main sections.\n* `clusters` - describe each cluster. You should set cluster alias as a name of the property of `clusters` object.\nPut all kafka properties as a fields of your alias.\n* `traffic` - an array that defines how data is copied between the clusters. For each flow\nset source cluster alias in `from` and target cluster alias in `target`\n\n`connectorName` should be the name of the connector with \n`\"connector.class\": \"org.apache.kafka.connect.mirror.MirrorSourceConnector\"`. This property will be used\nto collect offsets from the mm2 offsets topic.\n\n`mm2OffsetsTopic` - the name of the topic, used by mm2 to store offsets\n\n\n# Run application\n\nCheckout the sources:\n```shell\ngit clone https://github.com/papirosko/mm2-lag.git\ncd mm2-lag\n```\nCreate `clusters.yaml`.\n\nCompile and run the application:\n```shell\nsbt clean stage\n./target/universal/stage/bin/mm2-lag\n```\n\nOpen http://localhost:8080. Check [metrics](http://localhost:8080/metrics) endpoint, take a look at\n`counters` section, you should see `SourceConsumer` and `TargetConsumer` counters increasing for each cluster.\n\n\nAlso you can build archive with the application:\n```shell\nsbt clean universal:packageBin\n```\n\nYou can override the cluster file location with environment variable `CLUSTERS_FILE`:\n```shell\nCLUSTERS_FILE=\"/etc/clusters.yaml\" ./target/universal/stage/bin/mm2-lag\n```\n\n\n# Lag values\nUse `/api/lag_per_topic` endpoint to see lag values per each topic in text format.\n\nUse `/api/prometheus_lag` to see all lags per partition for all clusters. This endpoint can be used for prometheus \nto grab the metrics.\n\n\n\n# Docker\n\nBuild and run docker image locally:\n\nCreate `clusters.yaml`.\n\n```shell\nsbt docker:publishLocal\ndocker run --mount type=bind,source=$(pwd)/clusters.yaml,target=/opt/docker/clusters.yaml mm2-lag\n```\nYou can mount the required files in the same way, e.g. `*.jks` files.\n\n\nUse this to push image to remote registry:\n```shell\nDOCKER_REGISTRY=\"gitlab.server:5050\" sbt docker:publish\n```\n\n\n\nYou can also use docker image from docker.io: https://hub.docker.com/r/papirosko/mm2-lag\n\n\n\n# Development\n\nBuild and publish (any branch):\n```shell\nDOCKER_REGISTRY=\"docker.io\" CI_REGISTRY_IMAGE=\"docker.io/papirosko/mm2-lag\" sbt clean docker:publish\n```\n\nFor `main` branch it will deploy docker image with tag equal to the app version, so you need to increase app version\nbefore publishing. Also it will update `latest` tag.\n\nFor any other branch it will deploy docker image with the branch name as a tag.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpapirosko%2Fmm2-lag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpapirosko%2Fmm2-lag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpapirosko%2Fmm2-lag/lists"}