{"id":21423149,"url":"https://github.com/farbodahm/kafka-with-sample-data","last_synced_at":"2025-03-16T20:25:22.282Z","repository":{"id":263886783,"uuid":"891194985","full_name":"farbodahm/kafka-with-sample-data","owner":"farbodahm","description":"A complete Docker-based Kafka development environment with preloaded sample data. Includes configuration for brokers, schema registry, Kafka Connect, and sample connectors to kickstart local development and testing.","archived":false,"fork":false,"pushed_at":"2024-11-23T19:52:42.000Z","size":983,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-23T07:11:18.820Z","etag":null,"topics":["docker","kafka","kafka-connect","schema-registry"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/farbodahm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-19T22:32:27.000Z","updated_at":"2024-11-23T19:51:39.000Z","dependencies_parsed_at":"2024-11-20T20:37:06.829Z","dependency_job_id":"70e59cd1-9147-40b5-9be3-6d18cfdae9d8","html_url":"https://github.com/farbodahm/kafka-with-sample-data","commit_stats":null,"previous_names":["farbodahm/kafka-with-sample-data"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farbodahm%2Fkafka-with-sample-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farbodahm%2Fkafka-with-sample-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farbodahm%2Fkafka-with-sample-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farbodahm%2Fkafka-with-sample-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/farbodahm","download_url":"https://codeload.github.com/farbodahm/kafka-with-sample-data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243926480,"owners_count":20369990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","kafka","kafka-connect","schema-registry"],"created_at":"2024-11-22T21:14:29.379Z","updated_at":"2025-03-16T20:25:22.248Z","avatar_url":"https://github.com/farbodahm.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Quick POC Setup\n\nWhenever I wanted to do a quick POC with Kafka, I was spending an hour spinning up a Kafka broker with sample data. To address that, I created this Docker Compose setup so you can spin up a Kafka environment with sample data by simply running `docker compose up`.\n\nThis setup includes the following components:\n\n1. **Kafka Broker**: The core messaging system.\n2. **Schema Registry**: To manage Avro, Protobuf, and JSON schemas.\n3. **Kafka Connect**: Includes a pre-configured `DataGen` connector to generate sample data.\n4. **Kafka Control Center**: For managing and monitoring your Kafka cluster.\n\n## Features\n\n- **Quick Setup**: Bring up the entire Kafka environment in seconds.\n- **Sample Data Generation**: Automatically generate sample data using the DataGen connector with JSON, Avro, and Protobuf formats.\n- **Schema Support**: Full support for Avro, Protobuf, and JSON schemas via the Schema Registry.\n- **Easy Monitoring**: Monitor topics, consumer groups, and more using the Kafka Control Center.\n![AvailableConnectors](./assets/sample_data.png)\n\n---\n\n## Requirements\n\n- **Docker** and **Docker Compose** must be installed on your machine.\n\n---\n\n## Getting Started\n\n1. Clone this repository:\n    \n    ```bash\n    git clone git@github.com:farbodahm/kafka-with-sample-data.git\n    cd kafka-with-sample-data\n    ```\n    \n2. Start the environment:\n    \n    ```bash\n    docker compose up\n    ```\n    \n3. Access Kafka Control Center:\n    - Open your browser and go to `http://localhost:9021`\n    - Start the connector you want. This compose  will create 3 connectors for you:\n      - User sample data with **Json** schema\n      - User sample data with **Avro** schema\n      - User sample data with **Protobuf** schema\n      - **Nested** Order sample data with **Json** schema\n        - `Order` topic is joinable with `User` topic on `user_id` field.\n      ![AvailableConnectors](./assets/connectors.png)\n4. View and manage the generated data:\n    - Data will be generated on predefined topics by the DataGen connector.\n    - Check the topics in Kafka Control Center or using Kafka CLI tools.\n\n- Example User data:\n```json\n{\n  \"registertime\": \"1491454553823\",\n  \"userid\": \"User_6\",\n  \"regionid\": \"Region_1\",\n  \"gender\": \"OTHER\"\n}\n```\n\n- Example Order data:\n```json\n{\n  \"user_id\": \"User_6\",\n  \"order_id\": \"Order_318737\",\n  \"shipping_address\": {\n    \"city\": \"City_15\",\n    \"state\": \"State_3\",\n    \"zipcode\": 12130\n  },\n  \"items\": [\n    {\n      \"item_id\": \"Item_548\",\n      \"item_name\": \"  9o11\",\n      \"quantity\": 2072,\n      \"price\": 0.93797684\n    },\n    {\n      \"item_id\": \"Item_791\",\n      \"item_name\": \"I\",\n      \"quantity\": 1864,\n      \"price\": 0.88732815\n    },\n    {\n      \"item_id\": \"Item_455\",\n      \"item_name\": \"9F \",\n      \"quantity\": 4930,\n      \"price\": 0.60063416\n    }\n  ],\n  \"order_timestamp\": 315558009000,\n  \"order_status\": \"Shipped\"\n}\n```\n\n---\n\n## Customization\n\nYou can customize the setup by modifying the `docker-compose.yml` file and related connector configurations under `config` module:\n\n**Connector Configuration**: Update the DataGen connector configurations to generate data matching your POC requirements.\n\n- You can either use predefined schemas [here](https://github.com/confluentinc/kafka-connect-datagen/tree/master/config).\n- Or add you own schema to generate sample data based on your requirements following [here](https://github.com/confluentinc/kafka-connect-datagen/tree/master/config).\n\nThen you can easily add a new connector via:\n```bash\ncurl -X POST -H \"Content-Type: application/json\" --data @config/NEW_CONFIG.json http://localhost:8083/connectors\n```\n\n---\n\n## Stopping the Setup\n\nTo stop the Kafka environment, run:\n\n```bash\ndocker compose down\n```\n\n---\n\n## Troubleshooting\n\nIf you encounter issues, check the logs of individual services:\n\n```bash\ndocker logs \u003ccontainer-name\u003e\n```\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarbodahm%2Fkafka-with-sample-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffarbodahm%2Fkafka-with-sample-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarbodahm%2Fkafka-with-sample-data/lists"}