{"id":24772544,"url":"https://github.com/williamkoller/kafka-stream-processor","last_synced_at":"2026-04-13T01:11:32.914Z","repository":{"id":274665077,"uuid":"923654917","full_name":"williamkoller/kafka-stream-processor","owner":"williamkoller","description":"Kafka Stream Processor","archived":false,"fork":false,"pushed_at":"2025-01-28T16:56:12.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T17:34:44.040Z","etag":null,"topics":["docker","docker-compose","kafka","kafka-consumer","kafka-producer","kafka-streams","nodejs","typescript","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/williamkoller.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-28T16:27:42.000Z","updated_at":"2025-01-28T16:56:15.000Z","dependencies_parsed_at":"2025-01-28T17:44:49.473Z","dependency_job_id":null,"html_url":"https://github.com/williamkoller/kafka-stream-processor","commit_stats":null,"previous_names":["williamkoller/kafka-stream-processor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamkoller%2Fkafka-stream-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamkoller%2Fkafka-stream-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamkoller%2Fkafka-stream-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamkoller%2Fkafka-stream-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/williamkoller","download_url":"https://codeload.github.com/williamkoller/kafka-stream-processor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245168813,"owners_count":20571799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kafka-consumer","kafka-producer","kafka-streams","nodejs","typescript","zookeeper"],"created_at":"2025-01-29T04:23:17.877Z","updated_at":"2026-04-13T01:11:27.886Z","avatar_url":"https://github.com/williamkoller.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI/CD Pipeline](https://github.com/williamkoller/kafka-stream-processor/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/williamkoller/kafka-stream-processor/actions/workflows/ci-cd.yml)\n\n# Kafka Stream Processor\n\nKafka Stream Processor is a simple yet powerful application for producing and consuming messages using Apache Kafka. It includes logging functionality to track the flow of data and handle errors effectively.\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Features](#features)\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Running the Producer](#running-the-producer)\n  - [Running the Consumer](#running-the-consumer)\n- [Configuration](#configuration)\n- [Project Structure](#project-structure)\n\n## Introduction\n\nKafka Stream Processor is designed to simplify the process of sending and receiving messages using Apache Kafka. It supports message compression, error handling, and detailed logging to ensure reliability and traceability of the data flow.\n\n## Features\n\n- Produce and consume messages with Apache Kafka\n- Detailed logging for monitoring and debugging\n- Message compression using GZIP\n- Graceful shutdown and error handling\n\n## Prerequisites\n\nBefore you begin, ensure you have the following installed:\n\n- Node.js (v20 or higher)\n- Docker (for running Kafka and Zookeeper)\n\n## Installation\n\n1. Clone the repository:\n\n```sh\ngit clone https://github.com/williamkoller/kafka-stream-processor.git\ncd kafka-stream-processor\n```\n\n2. Install the dependencies:\n\n```sh\nnpm install\n```\n\n## Usage\n\n### Running the main\n\nTo run the main and send messages to the Kafka topic:\n\n1. Ensure Kafka and Zookeeper are running. You can use Docker Compose for convenience:\n\n```sh\ndocker-compose down -v \u0026\u0026 docker-compose up -d\n```\n\n2. Start:\n\n```sh\nnpm run start\n```\n\n### Configuration\n\nYou can configure the Kafka connection and other settings using environment variables. Create a `.env` file in the root directory with the following content:\n\n```env\nAPP_NAME=microservice-node\nBOOTSTRAP_SERVER=localhost:9092\nKAFKA_TOPIC=topic-test\n```\n\n## Project Structure\n\n```plaintext\nsrc\n├── infra\n│   ├── log\n│   │   └── index.ts\n│   └── streams\n│       └── kafka\n│           ├── consumer.ts\n│           ├── index.ts\n│           └── producer.ts\n└── main.ts\n```\n\n- infra/log: Contains the logging setup using a custom logger.\n- infra/streams/kafka: Contains the Kafka producer and consumer implementations.\n- main.ts: Entry point for the application, initializing the producer and consumer.\n\n# Example\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"./imgs/terminal.png\"\u003e\n    \u003cimg src=\"./imgs/terminal.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamkoller%2Fkafka-stream-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwilliamkoller%2Fkafka-stream-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamkoller%2Fkafka-stream-processor/lists"}