{"id":25171816,"url":"https://github.com/basemax/kafkadataprocessinggo","last_synced_at":"2026-05-14T21:03:34.639Z","repository":{"id":189730607,"uuid":"680284467","full_name":"BaseMax/KafkaDataProcessingGo","owner":"BaseMax","description":"Welcome to the Kafka Data Processing with Go project! This project showcases how to use Apache Kafka in combination with the Go programming language to build a data processing application.","archived":false,"fork":false,"pushed_at":"2024-03-13T22:59:23.000Z","size":46,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-28T01:38:17.604Z","etag":null,"topics":["ecommerce","go","go-grafana","go-kafka","golang-kafka","grafana","grafana-dashboard","grafana-dashboards","grafana-datasource","grafana-go","grafana-panel","kafka","kafka-client","kafka-connect","kafka-connector","kafka-consumer","kafka-manager","kafka-producer","kafka-streams"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BaseMax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-18T19:55:58.000Z","updated_at":"2024-03-22T17:30:06.000Z","dependencies_parsed_at":"2025-08-25T10:26:13.376Z","dependency_job_id":"70a253b2-0110-4ba4-8d9c-f5dd044e5159","html_url":"https://github.com/BaseMax/KafkaDataProcessingGo","commit_stats":null,"previous_names":["basemax/kafkadataprocessinggo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BaseMax/KafkaDataProcessingGo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BaseMax%2FKafkaDataProcessingGo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BaseMax%2FKafkaDataProcessingGo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BaseMax%2FKafkaDataProcessingGo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BaseMax%2FKafkaDataProcessingGo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BaseMax","download_url":"https://codeload.github.com/BaseMax/KafkaDataProcessingGo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BaseMax%2FKafkaDataProcessingGo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33043249,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ecommerce","go","go-grafana","go-kafka","golang-kafka","grafana","grafana-dashboard","grafana-dashboards","grafana-datasource","grafana-go","grafana-panel","kafka","kafka-client","kafka-connect","kafka-connector","kafka-consumer","kafka-manager","kafka-producer","kafka-streams"],"created_at":"2025-02-09T09:22:12.475Z","updated_at":"2026-05-14T21:03:34.610Z","avatar_url":"https://github.com/BaseMax.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Data Processing with Go\n\nWelcome to the Kafka Data Processing with Go project! This project showcases how to use Apache Kafka in combination with the Go programming language to build a data processing application. By following this example, you'll learn how to produce and consume data using Kafka topics, allowing you to develop scalable and efficient data processing pipelines.\n\nIn this project, we'll create a data processing application using the Go programming language and Apache Kafka. Imagine you're building a system that processes user activity data from a website and performs real-time analytics on it. Kafka will serve as the backbone for data streaming, enabling the efficient transfer of data between different components.\n\n## Prerequisites\n\nBefore you begin, make sure you have the following prerequisites:\n\n- Go programming language (Installation guide: Getting Started with Go)\n- Apache Kafka (Installation guide: Kafka Quickstart)\n- Git\n\n## Setup\n\nClone this repository:\n\n```bash\ngit clone https://github.com/basemax/KafkaDataProcessingGo.git\ncd KafkaDataProcessingGo\n```\n\nStart the Kafka server and create the necessary topics (assuming you've already installed Kafka):\n\n```bash\n# Start the ZooKeeper server (if not already started)\nbin/zookeeper-server-start.sh config/zookeeper.properties\n\n# Start the Kafka server\nbin/kafka-server-start.sh config/server.properties\n\n# Create the required topics\nbin/kafka-topics.sh --create --topic activities --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1\n```\n\nStart prometheus server and config like this:\n```yaml\nscrape_configs:\n  - job_name: KafkaProcessing\n    metrics_path: /metrics\n    honor_labels: false\n    honor_timestamps: true\n    scheme: http\n    scrape_interval: 1s\n    follow_redirects: true\n    body_size_limit: 0\n    sample_limit: 0\n    label_limit: 0\n    label_name_length_limit: 0\n    label_value_length_limit: 0\n    target_limit: 0\n    static_configs:\n      - targets:\n          - \"HOSTNAME:8000\"\n```\n\nYou can optionally using Grafana. Grafana configurations placed on `grafana/` directory.\n\n### Usage\n\nThis project consists of two main components: Producer and Consumer.\n\n### Run via docker\n\n#### Docker setup\n\nYou can change .env configurations.\n\n#### Start app\n\nStart application and it's dependencies using docker compose.\n```bash\nmake run\n```\nor\n```bash\ndocker-compose up -d\n```\n\nYou can check some endpoints to ensure checking health of system.\n - localhost:8000/metrics - Consumer application\n - localhost:29092 - Kafka server\n - localhost:9090 - Prometheus server\n - localhost:3000 - Grafana server\n\n#### Config Grafana\n\nImport datasource configurations using following command:\n```bash\nmake grafana_import_ds\n```\n\nImport Grafana dashboards manually. You can copy json configuration on `grafana/dashboards/` directory and import it to Grafana.\n\nNOTE: If you wanna build application image multiple times you can use `go mod vendor` command to keeping dependencies on container. With this technique build process will speed up.\n\n### Producer\n\nThe producer generates mock user activity data and sends it to the Kafka topic. To run the producer:\n\n```bash\ngo run . producer\n```\n\nUse `-f` option for creating fake delay on publishing Kafka messages.\n\nThe producer will continuously generate and send user activity data to the Kafka topic.\n\n### Consumer\n\nThe consumer subscribes to the Kafka topic, processes the user activity data, and performs analytics. To run the consumer:\n\n```bash\ngo run . consumer\n```\n\nThe consumer will listen for incoming user activity data and process it accordingly.\n\n### Faker\n\nThe faker create sample dataset for producer.\n\n```bash\ngo run . faker\n```\n\n## Sample Dataset\n\n\u003c!-- For your convenience, we've included a sample dataset in the sample_data directory. This dataset contains mock user activity logs that you can use to test the application. --\u003e\n\nThis application deals with tracking user activities on an e-commerce website. Here's a simple example of a JSON-based user activity dataset:\n\n```json\n[\n  {\n    \"user_id\": \"user123\",\n    \"timestamp\": \"2023-08-18T10:00:00Z\",\n    \"action\": \"view\",\n    \"product_id\": \"prod456\"\n  },\n  {\n    \"user_id\": \"user456\",\n    \"timestamp\": \"2023-08-18T11:30:00Z\",\n    \"action\": \"add_to_cart\",\n    \"product_id\": \"prod123\"\n  },\n  {\n    \"user_id\": \"user789\",\n    \"timestamp\": \"2023-08-18T12:15:00Z\",\n    \"action\": \"purchase\",\n    \"product_id\": \"prod789\"\n  },\n  // More entries...\n]\n```\n\nYou can create a sample dataset file like sample_data.json in the root directory of your project with multiple such entries.\n\nPlease note that this is just a basic representation, and you can extend it with additional fields and more complex data as needed for your application.\n\nFor generating a larger dataset, you might consider using libraries like Faker (for generating realistic fake data) in combination with Go's built-in JSON handling capabilities. Here's a rough example of how you could generate a larger dataset using Faker:\n\n```go\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/bxcodec/faker/v3\"\n)\n\ntype UserActivity struct {\n\tUserID     string    `json:\"user_id\"`\n\tTimestamp  time.Time `json:\"timestamp\"`\n\tAction     string    `json:\"action\"`\n\tProductID  string    `json:\"product_id\"`\n}\n\nfunc main() {\n\tvar activities []UserActivity\n\n\tfor i := 0; i \u003c 1000; i++ {\n\t\tactivity := UserActivity{\n\t\t\tUserID:     faker.UUIDHyphenated(),\n\t\t\tTimestamp:  faker.DateUnix(),\n\t\t\tAction:     faker.RandomChoice([]string{\"view\", \"add_to_cart\", \"purchase\"}),\n\t\t\tProductID:  faker.UUIDHyphenated(),\n\t\t}\n\t\tactivities = append(activities, activity)\n\t}\n\n\tfile, err := os.Create(\"sample_data.json\")\n\tif err != nil {\n\t\tfmt.Println(\"Error creating file:\", err)\n\t\treturn\n\t}\n\tdefer file.Close()\n\n\tencoder := json.NewEncoder(file)\n\tencoder.SetIndent(\"\", \"  \")\n\tif err := encoder.Encode(activities); err != nil {\n\t\tfmt.Println(\"Error encoding JSON:\", err)\n\t\treturn\n\t}\n\n\tfmt.Println(\"Sample data generated and saved to sample_data.json\")\n}\n```\n\nRemember that this is just a basic example to get you started. Depending on your application's needs, you might want to generate more complex data with a wider range of possible actions, timestamps, and user profiles.\n\n## Contributing\n\nContributions are welcome! If you encounter any issues or want to add new features, feel free to open a pull request. For significant changes, please open an issue first to discuss your proposed changes.\n\n## License\n\nThis project is licensed under the GPL-3.0 License - see the LICENSE file for details.\n\nCopyright 2023, Max Base\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasemax%2Fkafkadataprocessinggo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbasemax%2Fkafkadataprocessinggo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasemax%2Fkafkadataprocessinggo/lists"}