{"id":23623595,"url":"https://github.com/freedisch/kafka-go","last_synced_at":"2025-11-07T08:30:28.417Z","repository":{"id":217738810,"uuid":"741647030","full_name":"Freedisch/kafka-go","owner":"Freedisch","description":"These projects address real-world scenarios, promoting concurrency, scalability, and efficient integration with messaging systems.","archived":false,"fork":false,"pushed_at":"2024-01-11T22:57:00.000Z","size":22,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-27T20:49:57.533Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Freedisch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2024-01-10T20:28:24.000Z","updated_at":"2024-03-19T17:14:30.000Z","dependencies_parsed_at":"2024-01-18T04:23:31.694Z","dependency_job_id":"c66a6bca-5c15-4e05-b0e5-5da656fb2fdd","html_url":"https://github.com/Freedisch/kafka-go","commit_stats":null,"previous_names":["freedisch/kafka-go"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freedisch%2Fkafka-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freedisch%2Fkafka-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freedisch%2Fkafka-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freedisch%2Fkafka-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Freedisch","download_url":"https://codeload.github.com/Freedisch/kafka-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239523847,"owners_count":19653020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-27T20:49:42.847Z","updated_at":"2025-11-07T08:30:28.365Z","avatar_url":"https://github.com/Freedisch.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Webserver with Messaging System Integration\n\n## Problem Statement\n\nThe task is to build a web server in Go to handle incoming data and route it through messaging systems. The web server should have two endpoints: one to receive data via POST requests and another to retrieve data via GET requests.\n\n### Specifications:\n\n1. **POST Endpoint:**\n   - Receives incoming data through a POST request.\n   - Utilizes Goroutine-1 to route the data to a Kafka topic.\n   - Another Goroutine-2 listens to this Kafka topic and routes the data to a Redis database.\n\n2. **GET Endpoint:**\n   - Retrieves data from Redis through a GET request.\n   - Uses Goroutine-3 to send the data to Kafka and deliver the response.\n\n## Design of Request/Response\n\n- **POST Request Flow:**\n  1. Client sends a POST request to the designated endpoint.\n  2. Web server handles the request, spawns Goroutine-1 to send data to Kafka topic.\n  3. Goroutine-2 picks up the data from the Kafka topic and routes it to Redis.\n\n- **GET Request Flow:**\n  1. Client sends a GET request to the designated endpoint.\n  2. Web server retrieves data from Redis using Goroutine-3.\n  3. The retrieved data is sent to Kafka for further processing, and the response is delivered to the client.\n\n## Implementation Details\n\n- Use a local Docker-based Kafka system for testing and development.\n- Utilize the Sarama library, a reliable Kafka client for Go.\n- Ensure that transactions are non-blocking to prevent issues with Golang concurrency.\n\n## Setting Up the Development Environment\n\n1. Install Docker: [Docker Installation Guide](https://docs.docker.com/get-docker/)\n2. Set up a local Kafka system using Docker.\n   ```bash\n   docker run -d --name kafka -p 9092:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT wurstmeister/kafka\n   ```\n3. Use the Sarama library for Go. Install it using:\n   ```bash\n   go get -u github.com/IBM/sarama\n   ```\n\n## Notes and Considerations\n\n- Sarama is recommended for Kafka integration due to its reliability and widespread use in the Go community.\n- Transactions in Golang should be non-blocking to maintain the efficiency of the web server.\n\n---\n\n# Simple Scalable Webserver\n\n## Problem Statement\n\nBuild a simple web server in Go that scales as it receives more requests. The server should spawn more Goroutines as the same endpoint handles numerous requests.\n\n### Specifications:\n\n- **Endpoint:**\n  - The application should have only one endpoint.\n  - Introduce a sleep operation to simulate a compute-heavy operation.\n  - The server should scale by spawning additional Goroutines as requests increase within the sleep interval.\n\n## Solution Approach\n\n- Study and implement a mechanism to dynamically scale Goroutines based on incoming requests.\n- Introduce sleep operations to simulate compute-heavy tasks and test the scalability of the server.\n- Design the system to handle an increasing number of requests efficiently.\n\n## Implementation Details\n\n- Single endpoint `/process` for handling compute-heavy tasks.\n- Worker pool for efficient concurrency control.\n- Simulates client sending requests to the server.\n- Demonstrates basic scalability by adjusting the number of workers.\n\n## Prerequisites\n\n- Go installed on your machine.\n\n## Getting Started\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/freedisch/kafka-go.git\n   ```\n\n2. Change into the project directory:\n\n   ```bash\n   cd kafka-go\n   ```\n\n3. Run the server:\n\n   ```bash\n   go run main.go\n   ```\n\n   The server will start at http://localhost:8080.\n\n4. Simulate client requests:\n\n   Open a new terminal and run:\n\n   ```bash\n   go run client.go\n   ```\n\n   This will simulate multiple client requests to the server.\n\n## Configuration\n\nAdjust the `workerCount` variable in `main.go` to control the number of concurrent workers in the pool.\n\n```go\nvar (\n\tworkerCount = 5\n)\n```\n\n## Contributing\n\nFeel free to contribute to this project by opening issues or submitting pull requests. Your feedback and contributions are welcome!\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedisch%2Fkafka-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedisch%2Fkafka-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedisch%2Fkafka-go/lists"}