https://github.com/dfarr/kafkanaut
Argo Events Sensor POC
https://github.com/dfarr/kafkanaut
argo argo-events kafka pulsar
Last synced: 11 days ago
JSON representation
Argo Events Sensor POC
- Host: GitHub
- URL: https://github.com/dfarr/kafkanaut
- Owner: dfarr
- Created: 2022-12-01T22:07:34.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-01-06T23:11:55.000Z (over 2 years ago)
- Last Synced: 2025-03-26T15:42:15.720Z (28 days ago)
- Topics: argo, argo-events, kafka, pulsar
- Language: Go
- Homepage: https://dfarr.medium.com/exploring-messaging-systems-with-argo-events-a259f663bd30
- Size: 46.9 KB
- Stars: 5
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kafkanaut
An exploration of messaging systems with [Argo Events](https://argoproj.github.io/argo-events/).
Implements two alternative EventBus technologies: [Kafka](https://kafka.apache.org/) and [Pulsar](https://pulsar.apache.org/). Both implementations enable horizontal scaling of Argo Event's Sensor applications, something not currently possible with vanilla Argo Events.
## Architecture
## Setup
### Slack
A slack incoming webhook is required to run both the Kafka and Pulsar implementations. Follow [these instructions](https://slack.com/help/articles/115005265063-Incoming-webhooks-for-Slack) to set up a custom slack application with a webhook. Set the webhook URL as the `SLACK` environment variable when running the go program as described below.
### Kafka
To run the Kafka implementation you will need to have a [local broker](https://kafka.apache.org/quickstart) running on port 9092. The `{event, trigger, action}` topics will be automatically created if they do not exist. To play around with a different number of partitions you can run the following commands:
```
kafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic event
kafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic trigger
kafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic action
```
### Pulsar
To run the Pulsar implementation you will need to have a [local broker](https://pulsar.apache.org/docs/2.10.x/getting-started-standalone/) running on port 6650. The `{event, trigger, action}` topics will be automatically created if they do not exist, but as non-partitioned topics. To create partitioned topics (as intended) you can run the following comands:
```
bin/pulsar-admin topics create-partitioned-topic -p 3 event
bin/pulsar-admin topics create-partitioned-topic -p 3 trigger
bin/pulsar-admin topics create-partitioned-topic -p 3 action
```
### Sample Messages
```json
{"specversion": "1.0", "id": "1", "source": "es-1", "subject": "blue", "data": "blue"}
{"specversion": "1.0", "id": "2", "source": "es-2", "subject": "yellow", "data": "yellow"}
{"specversion": "1.0", "id": "3", "source": "es-3", "subject": "red", "data": "red"}
```
## Run
### Kafka
```
EB=kafka SLACK=https://hooks.slack.com/services/xxx go run ./...
```
Multiple instances can be run simultaneously. Run the following command to produce test mesages:
```
kafka-console-producer --bootstrap-server localhost:9092 --topic event
> {"specversion": "1.0", "id": "1", "source": "es-1", "subject": "blue", "data": "blue"}
> {"specversion": "1.0", "id": "2", "source": "es-2", "subject": "yellow", "data": "yellow"}
> {"specversion": "1.0", "id": "3", "source": "es-3", "subject": "red", "data": "red"}
```
### Pulsar
```
EB=pulsar SLACK=https://hooks.slack.com/services/xxx go run ./...
```
Multiple instances can be run simultaneously. Run the following commands to produce test mesages:
```
bin/pulsar-client produce event -s ,, -m '{"specversion": "1.0", "id": "1", "source": "es-1", "subject": "blue", "data": "blue"}'
bin/pulsar-client produce event -s ,, -m '{"specversion": "1.0", "id": "2", "source": "es-2", "subject": "yellow", "data": "yellow"}'
bin/pulsar-client produce event -s ,, -m '{"specversion": "1.0", "id": "3", "source": "es-3", "subject": "red", "data": "red"}'
```