{"id":17292827,"url":"https://github.com/lesovsky/kafka-poc","last_synced_at":"2026-02-02T17:12:17.183Z","repository":{"id":69264871,"uuid":"435153418","full_name":"lesovsky/kafka-poc","owner":"lesovsky","description":"Kafka cluster PoC","archived":false,"fork":false,"pushed_at":"2021-12-05T11:59:27.000Z","size":6696,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-27T07:03:19.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lesovsky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-05T11:56:28.000Z","updated_at":"2021-12-29T23:40:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"4bf7d2e5-c3e9-46f5-a30d-1d28b46a4a5f","html_url":"https://github.com/lesovsky/kafka-poc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lesovsky/kafka-poc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lesovsky%2Fkafka-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lesovsky%2Fkafka-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lesovsky%2Fkafka-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lesovsky%2Fkafka-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lesovsky","download_url":"https://codeload.github.com/lesovsky/kafka-poc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lesovsky%2Fkafka-poc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259768159,"owners_count":22908209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T10:44:29.273Z","updated_at":"2026-02-02T17:12:17.084Z","avatar_url":"https://github.com/lesovsky.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka cluster PoC\n\n## Index of content\n- [Task](#task)\n- [Notes](#notes)\n- [Setup and usage](#setup-and-usage)\n\n### Task\nWe need to test and prepare a proof-of-concept Kafka setup. Requirements are as follow:\n\nIt should be reliable (survive at least 1 node loss).\nWe should minimise latency between read and write operations as much as possible (so consumer should be able to read message as fast as possible after producer publish it)\nAt the end of the task we would like to have a document with a detailed description of setup.\n\n### Notes\n- Use [docker-compose](https://docs.docker.com/compose/install/) as a quick and simple tool for demo-PoCs\n- Use Zookeeper instead of KRaft (due to Zookeeper maturity) \n- Use Zookeeper HA configuration for fault tolerance\n- Use Kafka HA configuration for fault tolerance\n- Kafka containers expose the 29093, 29094, 29095 ports, and they have to be free.   \n- Grafana container exposes the 3000 port, and it has to be free.   \n- Use `make` for basic setup operations and [trubka](https://github.com/xitonix/trubka) for generating test workload\n- Use simple Prometheus/Grafana setup for collecting and visualizing metrics (with no detailed dashboards)\n\n### Implementation\n\u003e It should be reliable (survive at least 1 node loss).\n\nReliability is achieved for particular topics. When topic is created its replication factor has to be specified. Default replication factor could be\nalso configured for auto-created topics in server configuration. Test topic used here is created with replication factor 3. \n\n*A common production setting is a replication factor of 3, i.e., there will always be three copies of your data. This replication is performed at the level of topic-partitions.* [Kafka docs](https://kafka.apache.org/documentation/#intro_concepts_and_terms).\n\n\u003e We should minimise latency between read and write operations as much as possible (so consumer should be able to read message\nas fast as possible after producer publish it)*\n\nKafka default configuration is optimized for latency and almost all settings don't need to adjust.\nAt the same time, read/write latencies depend on number of partitions of topics. Due to this, test topic is created as a \nsingle partition to minimize latencies. All Kafka instances are running with min.insync.replicas: 1, which should improve \nproducers latencies.\n\n\u003e At the end of the task we would like to have a document with a detailed description of setup.\n\n### Setup and usage\nThe following tools are required and have to be present on system: `docker-compose`, `make`. \nMakefile is a wrapper over `docker-compose` and `trubka` utilities, it is possible to execute them directly for custom operations.  \n\n1. Up the environment.\n```shell\nmake docker/up\n```\n\n2. Create a test topic.\n```shell\nmake kafka/create-topic\n```\n\n3. Start to consume from test topic.\n```shell\nmake kafka/consume\n```\n\n4. Open a new terminal and start to produce messages into test topic.\n```shell\nmake kafka/produce\n```\n\n5. Open a grafana [dashboard](http://localhost:3000/d/uxjb-Shnz/kafka-overview?orgId=1) in a browser (login: `admin` password: `password`)\n\n\n6. Restart a random container with Kafka\n```shell\nmake docker/restart/kafka\n```\n\n7. Press Ctrl+C to interrupt produce/consume processes.\n\n\n8. Down the environment\n```shell\nmake docker/down\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flesovsky%2Fkafka-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flesovsky%2Fkafka-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flesovsky%2Fkafka-poc/lists"}