Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dalelane/eventstreams-mirrormaker2-demos
Using Mirror Maker 2 with IBM Event Streams
https://github.com/dalelane/eventstreams-mirrormaker2-demos
kafka mirrormaker mm2
Last synced: 24 days ago
JSON representation
Using Mirror Maker 2 with IBM Event Streams
- Host: GitHub
- URL: https://github.com/dalelane/eventstreams-mirrormaker2-demos
- Owner: dalelane
- License: apache-2.0
- Created: 2024-03-29T15:58:35.000Z (7 months ago)
- Default Branch: master
- Last Pushed: 2024-07-26T12:40:49.000Z (3 months ago)
- Last Synced: 2024-07-27T10:46:10.812Z (3 months ago)
- Topics: kafka, mirrormaker, mm2
- Language: Shell
- Homepage: https://dalelane.co.uk/blog/?p=5098
- Size: 23.5 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Demos of Mirror Maker 2 with IBM Event Streams
Mirror Maker 2 is a powerful and flexible tool for moving Kafka events between Kafka clusters. These demos aim to inspire you to think about different ways that you could use it in your own projects.
A great way to learn how to use it is to play with it for yourself, so this repo includes Ansible playbooks that create examples of a variety of different Mirror Maker topologies.
These demonstrations all run in a single OpenShift cluster, using separate namespaces to represent "regions".
To see the requirements for creating these demos, see [REQUIREMENTS.md](./REQUIREMENTS.md).
A more detailed description of what these demos illustrate can be found in the blog posts starting from https://dalelane.co.uk/blog/?p=5098
## Using Mirror Maker 2 to aggregate events from multiple regions
Ansible playbook: [`01-aggregate-central/setup.yaml`](./01-aggregate-central/setup.yaml)
Example of how to run the playbook: [`setup-01-aggregate-central.sh`](./setup-01-aggregate-central.sh)
To see the events received by the consumer application running in the "Europe region", you can run the [`consumer-europe.sh`](./consumer-europe.sh) script.
#### An alternative topology:
Ansible playbook: [`02-aggregate-distributed/setup.yaml`](./02-aggregate-distributed/setup.yaml)
Example of how to run the playbook: [`setup-02-aggregate-distributed.sh`](./setup-02-aggregate-distributed.sh)
## Using Mirror Maker 2 to broadcast events to multiple regions
Ansible playbook: [`03-broadcast/setup.yaml`](./03-broadcast/setup.yaml)
Example of how to run the playbook: [`setup-03-broadcast.sh`](./setup-03-broadcast.sh)
To see the events received by the consumer applications, you can run the [`consumer-northamerica.sh`](./consumer-northamerica.sh) and [`consumer-southamerica.sh`](./consumer-southamerica.sh) scripts.
## Using Mirror Maker 2 to distribute topics across multiple regions
Ansible playbook: [`04-shared-aggregate/setup.yaml`](./04-shared-aggregate/setup.yaml)
Example of how to run the playbook: [`setup-04-shared-aggregate.sh`](./setup-04-shared-aggregate.sh)
To see the events received by the consumer applications, you can run the consumer scripts:
- [`consumer-northamerica.sh`](./consumer-northamerica.sh)
- [`consumer-southamerica.sh`](./consumer-southamerica.sh)
- [`consumer-europe.sh`](./consumer-europe.sh)
## Using Mirror Maker 2 to create a failover cluster
Ansible playbooks:
- [`05-shared-aggregate/setup.yaml`](./05-active-passive/setup.yaml)
- [`05-shared-aggregate/failover.yaml`](./05-active-passive/failover.yaml)
Example of how to run the playbooks:
- [`setup-05-active-passive.sh`](./setup-05-active-passive.sh)
- [`setup-05-failover-to-passive.sh`](./setup-05-failover-to-passive.sh)
## Using Mirror Maker 2 to restore events from a backup cluster
Ansible playbooks:
- [`06-backup-restore/initial-setup.yaml`](./06-backup-restore/initial-setup.yaml)
- [`06-backup-restore/recreate-from-backup.yaml`](./06-backup-restore/recreate-from-backup.yaml)
- [`06-backup-restore/resume-applications.yaml`](./06-backup-restore/resume-applications.yaml)
- [`06-backup-restore/setup-new-backup.yaml`](./06-backup-restore/setup-new-backup.yaml)
Example of how to run the playbooks:
- [`setup-06-backup.sh`](./setup-06-backup.sh)
- [`setup-06-restore.sh`](./setup-06-restore.sh)
- [`setup-06-resume.sh`](./setup-06-resume.sh)
- [`setup-06-setup-new-backup.sh`](./setup-06-setup-new-backup.sh)
## Using Mirror Maker 2 to migrate a Kafka cluster to a new region
Ansible playbooks:
- [`07-migration/initial-setup.yaml`](./07-migration/initial-setup.yaml)
- [`07-migration/migrate-topics.yaml`](./07-migration/migrate-topics.yaml)
- [`07-migration/pause-producers.yaml`](./07-migration/pause-producers.yaml)
- [`07-migration/migrate-consumers.yaml`](./07-migration/migrate-consumers.yaml)
- [`07-migration/migrate-producers.yaml`](./07-migration/migrate-producers.yaml)
- [`07-migration/cleanup.yaml`](./07-migration/cleanup.yaml)
Example of how to run the playbooks:
- [`setup-07-migration.sh`](./setup-07-migration.sh)
- [`setup-07-migrate-topics.sh`](./setup-07-migrate-topics.sh)
- [`setup-07-pause-producers.sh`](./setup-07-pause-producers.sh)
- [`setup-07-migrate-consumers.sh`](./setup-07-migrate-consumers.sh)
- [`setup-07-migrate-producers.sh`](./setup-07-migrate-producers.sh)
- [`setup-07-cleanup.sh`](./setup-07-cleanup.sh)