{"id":17084171,"url":"https://github.com/dfarr/kafkanaut","last_synced_at":"2025-04-12T21:25:11.768Z","repository":{"id":151900173,"uuid":"573185060","full_name":"dfarr/kafkanaut","owner":"dfarr","description":"Argo Events Sensor POC","archived":false,"fork":false,"pushed_at":"2023-01-06T23:11:55.000Z","size":48,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T15:42:15.720Z","etag":null,"topics":["argo","argo-events","kafka","pulsar"],"latest_commit_sha":null,"homepage":"https://dfarr.medium.com/exploring-messaging-systems-with-argo-events-a259f663bd30","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dfarr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-01T22:07:34.000Z","updated_at":"2023-06-22T09:44:36.000Z","dependencies_parsed_at":"2023-05-15T00:15:10.703Z","dependency_job_id":null,"html_url":"https://github.com/dfarr/kafkanaut","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfarr%2Fkafkanaut","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfarr%2Fkafkanaut/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfarr%2Fkafkanaut/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfarr%2Fkafkanaut/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dfarr","download_url":"https://codeload.github.com/dfarr/kafkanaut/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248633228,"owners_count":21136834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argo","argo-events","kafka","pulsar"],"created_at":"2024-10-14T13:06:01.301Z","updated_at":"2025-04-12T21:25:11.746Z","avatar_url":"https://github.com/dfarr.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafkanaut\nAn exploration of messaging systems with [Argo Events](https://argoproj.github.io/argo-events/).\n\nImplements two alternative EventBus technologies: [Kafka](https://kafka.apache.org/) and [Pulsar](https://pulsar.apache.org/). Both implementations enable horizontal scaling of Argo Event's Sensor applications, something not currently possible with vanilla Argo Events.\n\n## Architecture\n![architecture](https://user-images.githubusercontent.com/1387834/208555850-fcf1fd54-656d-4d91-a5c8-2d0497f9aa47.png)\n\n## Setup\n### Slack\nA slack incoming webhook is required to run both the Kafka and Pulsar implementations. Follow [these instructions](https://slack.com/help/articles/115005265063-Incoming-webhooks-for-Slack) to set up a custom slack application with a webhook. Set the webhook URL as the `SLACK` environment variable when running the go program as described below.\n\n### Kafka\nTo run the Kafka implementation you will need to have a [local broker](https://kafka.apache.org/quickstart) running on port 9092. The `{event, trigger, action}` topics will be automatically created if they do not exist. To play around with a different number of partitions you can run the following commands:\n```\nkafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic event\nkafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic trigger\nkafka-topics --bootstrap-server localhost:9092 --create --partitions 3 --topic action\n```\n\n### Pulsar\nTo run the Pulsar implementation you will need to have a [local broker](https://pulsar.apache.org/docs/2.10.x/getting-started-standalone/) running on port 6650. The `{event, trigger, action}` topics will be automatically created if they do not exist, but as non-partitioned topics. To create partitioned topics (as intended) you can run the following comands:\n```\nbin/pulsar-admin topics create-partitioned-topic -p 3 event\nbin/pulsar-admin topics create-partitioned-topic -p 3 trigger\nbin/pulsar-admin topics create-partitioned-topic -p 3 action\n```\n\n### Sample Messages\n```json\n{\"specversion\": \"1.0\", \"id\": \"1\", \"source\": \"es-1\", \"subject\": \"blue\", \"data\": \"blue\"}\n{\"specversion\": \"1.0\", \"id\": \"2\", \"source\": \"es-2\", \"subject\": \"yellow\", \"data\": \"yellow\"}\n{\"specversion\": \"1.0\", \"id\": \"3\", \"source\": \"es-3\", \"subject\": \"red\", \"data\": \"red\"}\n```\n\n## Run\n### Kafka\n```\nEB=kafka SLACK=https://hooks.slack.com/services/xxx go run ./...\n```\n\nMultiple instances can be run simultaneously. Run the following command to produce test mesages:\n```\nkafka-console-producer --bootstrap-server localhost:9092 --topic event\n\u003e {\"specversion\": \"1.0\", \"id\": \"1\", \"source\": \"es-1\", \"subject\": \"blue\", \"data\": \"blue\"}\n\u003e {\"specversion\": \"1.0\", \"id\": \"2\", \"source\": \"es-2\", \"subject\": \"yellow\", \"data\": \"yellow\"}\n\u003e {\"specversion\": \"1.0\", \"id\": \"3\", \"source\": \"es-3\", \"subject\": \"red\", \"data\": \"red\"}\n```\n\n### Pulsar\n```\nEB=pulsar SLACK=https://hooks.slack.com/services/xxx go run ./...\n```\n\nMultiple instances can be run simultaneously. Run the following commands to produce test mesages:\n```\nbin/pulsar-client produce event -s ,, -m '{\"specversion\": \"1.0\", \"id\": \"1\", \"source\": \"es-1\", \"subject\": \"blue\", \"data\": \"blue\"}'\nbin/pulsar-client produce event -s ,, -m '{\"specversion\": \"1.0\", \"id\": \"2\", \"source\": \"es-2\", \"subject\": \"yellow\", \"data\": \"yellow\"}'\nbin/pulsar-client produce event -s ,, -m '{\"specversion\": \"1.0\", \"id\": \"3\", \"source\": \"es-3\", \"subject\": \"red\", \"data\": \"red\"}'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfarr%2Fkafkanaut","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdfarr%2Fkafkanaut","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfarr%2Fkafkanaut/lists"}