{"id":28758793,"url":"https://github.com/rafi021/kafka-docker-compose","last_synced_at":"2026-04-16T10:34:51.407Z","repository":{"id":293028654,"uuid":"981723352","full_name":"rafi021/kafka-docker-compose","owner":"rafi021","description":"Docker Composer File for Kafka and Test using Express JS","archived":false,"fork":false,"pushed_at":"2025-05-13T09:50:36.000Z","size":7594,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-13T10:44:13.563Z","etag":null,"topics":["apache-kafka","docker","docker-compose","docker-container","expressjs","kafka","kafka-client","message-broker","pub-sub","queue"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rafi021.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-11T18:43:59.000Z","updated_at":"2025-05-13T09:50:40.000Z","dependencies_parsed_at":"2025-05-13T10:54:54.195Z","dependency_job_id":null,"html_url":"https://github.com/rafi021/kafka-docker-compose","commit_stats":null,"previous_names":["rafi021/kafka-docker-compose"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rafi021/kafka-docker-compose","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafi021%2Fkafka-docker-compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafi021%2Fkafka-docker-compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafi021%2Fkafka-docker-compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafi021%2Fkafka-docker-compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rafi021","download_url":"https://codeload.github.com/rafi021/kafka-docker-compose/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafi021%2Fkafka-docker-compose/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260288454,"owners_count":22986669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","docker","docker-compose","docker-container","expressjs","kafka","kafka-client","message-broker","pub-sub","queue"],"created_at":"2025-06-17T04:09:48.502Z","updated_at":"2026-04-16T10:34:51.392Z","avatar_url":"https://github.com/rafi021.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Docker Compose and Test with Express.js Application\n\n# What is Kafka?\n[![what is kafka](https://miro.medium.com/v2/resize:fit:1200/1*5EJ12zSrJCiY2BFdvGFNzg.gif)](https://www.youtube.com/shorts/Bc6oyURRIxQ?feature=share)\n\nThis project sets up a Kafka environment using Docker and Docker Compose, along with a Node.js Express application (`kafka-express-app`) to test Kafka by producing and consuming messages.\n\n---\n\n## Prerequisites\n\n- Docker and Docker Compose installed on your system.\n- Node.js and npm installed for running the Express.js application.\n\n---\n\n## Project Structure\n\n```\nkafka-docker/\n├── docker-compose.yml       # Docker Compose file for Kafka setup\n├── kafka-express-app/       # Node.js Express application\n│   ├── src/\n│   │   ├── kafka/\n│   │   │   ├── producer.ts  # Kafka producer logic\n│   │   │   └── consumer.ts  # Kafka consumer logic\n│   │   ├── routes/\n│   │   │   └── index.ts     # Express routes\n│   │   └── app.ts           # Express app entry point\n│   ├── package.json         # Node.js dependencies and scripts\n│   ├── tsconfig.json        # TypeScript configuration\n│   └── .env                 # Environment variables\n└── README.md                # Project documentation\n```\n\n---\n\n## Kafka Setup\n\n### Services in `docker-compose.yml`\n\n- **Zookeeper**: Manages Kafka brokers.\n- **Kafka Broker**: The Kafka server.\n- **Schema Registry**: Manages Avro schemas for Kafka topics.\n- **Kafka Connect**: For connecting Kafka to external systems.\n- **Control Center**: Web UI for managing Kafka.\n- **kafka-express-app**: Node.js application for testing Kafka.\n\n### Steps to Start Kafka\n\n1. **Start Docker Services**:\n   Run the following command to start all services:\n   ```bash\n   docker-compose up -d\n   ```\n\n2. **Verify Services**:\n   Check if all services are running:\n   ```bash\n   docker ps\n   ```\n\n3. **Access Control Center**:\n   Open the Confluent Control Center in your browser:\n   ```\n   http://localhost:9021\n   ```\n\n---\n\n## Kafka-Express-App Setup\n\n### Environment Variables\n\nCreate a `.env` file in the `kafka-express-app` directory with the following content:\n\n```env\nKAFKA_CLIENT_ID=my-app\nKAFKA_BROKERS=localhost:9092\nKAFKA_GROUP_ID=test-group\nKAFKA_TOPIC=test-topic\nKAFKA_AUTH_ENABLED=false\n```\n\nIf you connect to a secured Kafka cluster, set `KAFKA_AUTH_ENABLED=true` and provide `KAFKA_USERNAME`, `KAFKA_PASSWORD`, and optionally `KAFKA_SSL=true`.\n\n### Install Dependencies\n\nNavigate to the `kafka-express-app` directory and install the required dependencies:\n\n```bash\ncd kafka-express-app\nnpm install\n```\n\n### Build the Application\n\nCompile the TypeScript code to JavaScript:\n\n```bash\nnpm run build\n```\n\n### Run the Application\n\nStart the application in development mode:\n\n```bash\nnpm run dev\n```\n\nOr start the compiled application:\n\n```bash\nnpm start\n```\n\n---\n\n## Testing Kafka\n\n### Produce a Message\n\nUse a tool like `curl` or Postman to send a POST request to the `/api/publish` endpoint:\n\n```bash\ncurl -X POST http://localhost:3000/api/publish \\\n-H \"Content-Type: application/json\" \\\n-d '{\"topic\": \"test-topic\", \"message\": \"Hello Kafka!\"}'\n```\n\n### Consume Messages\n\nSend a GET request to the `/api/subscribe` endpoint to start consuming messages:\n\n```bash\ncurl http://localhost:3000/api/subscribe\n```\n\nYou should see the consumed messages logged in the console.\n\n---\n\n## Stopping Services\n\nTo stop all Docker services, run:\n\n```bash\ndocker-compose down\n```\n\n---\n\n## Troubleshooting\n\n- **Kafka Broker Connection Issues**:\n   Ensure the `KAFKA_BROKERS` in `.env` matches the broker address in `docker-compose.yml` (for local Docker Compose, use `localhost:9092` from your app host or `broker:29092` from another container). Leave `KAFKA_AUTH_ENABLED=false` unless the broker actually requires SASL authentication.\n\n- **Port Conflicts**:\n  Ensure the ports defined in `docker-compose.yml` (e.g., `9092`, `29092`, `9021`) are not in use by other applications.\n\n---\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafi021%2Fkafka-docker-compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafi021%2Fkafka-docker-compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafi021%2Fkafka-docker-compose/lists"}