Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shkvik/nestjs-kafka-producer
This is a NestJS application developed as a producer for Apache Kafka. It provides a simple and efficient interface for sending messages in the Kafka topic, supporting asynchronous operations and flexible configuration of the microservice architecture.
https://github.com/shkvik/nestjs-kafka-producer
broker kafka-producer nestjs
Last synced: 3 months ago
JSON representation
This is a NestJS application developed as a producer for Apache Kafka. It provides a simple and efficient interface for sending messages in the Kafka topic, supporting asynchronous operations and flexible configuration of the microservice architecture.
- Host: GitHub
- URL: https://github.com/shkvik/nestjs-kafka-producer
- Owner: shkvik
- License: mit
- Created: 2024-09-15T17:34:52.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-10-24T11:42:07.000Z (3 months ago)
- Last Synced: 2024-10-24T13:55:07.805Z (3 months ago)
- Topics: broker, kafka-producer, nestjs
- Language: TypeScript
- Homepage:
- Size: 495 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
[circleci-url]: https://circleci.com/gh/nestjs/nest
A progressive Node.js framework for building efficient and scalable server-side applications.
## Kafka Producer Template
This repository serves as a template for a Kafka producer, offering flexible and powerful ways to send messages to Kafka topics. It supports three distinct methods of sending messages, making it a versatile solution for various data processing scenarios.
## Features
1. **Broadcast**:
The message is sent to all partitions of a Kafka topic without binding to a specific key. This is used when you need to send a message to all consumers without partitioning by queues.
2. **Queueing by Key**:
The message is sent with a specific key, ensuring its delivery to a designated partition of the topic. This ensures sequential processing of messages with the same key and is used to manage tasks that require ordered processing in a queue.
3. **Rate-limited Sending**:
The ability to send messages at a specified rate limit. This approach is useful when you need to control the load on Kafka brokers or limit the processing speed for an even distribution of the load.
## How to Use
Clone the repository and configure the connection to your Kafka broker.
Choose and configure the desired method for sending messages (broadcast, queueing by key, or rate-limited sending).
Integrate the template into your application for quick and efficient Kafka message handling.
## BroadCast
Each application instance should have its own static consumer group, ensuring they can read from a single partition independently. This design allows for scalability by increasing the number of partitions if needed, though it's not a strict requirement for functionality.
```mermaid
flowchart TD
P["Producer"] --> K["Kafka"]
K -->|group Id 1| A["Consumer replica 1"]
K -->|group Id 2| B["Consumer replica 2"]
K -->|group Id 3| C["Consumer replica 3"]
```
## Queueing by Key
...
## Project setup
1. Install dependencies
```bash
$ npm install
```
2. Create .env file by .env.example
```env
KAFKAJS_NO_PARTITIONER_WARNING=1
NODE_ENV=local
HOST=localhost
...
```
## Compile and run the project
```bash
# development
$ npm run start
# watch mode
$ npm run start:dev
# production mode
$ npm run start:prod
```
## Run tests
```bash
# unit tests
$ npm run test
# e2e tests
$ npm run test:e2e
# test coverage
$ npm run test:cov
```
## Docker build & run
```bash
# build docker image
$ docker build -t nestjs-kafka-producer .
# run image with .env file
$ docker run --env-file .env -d -p 3000:3000 nestjs-kafka-producer
```
## Resources
Check out a few resources that may come in handy when working with NestJS:
- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
## Support
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
## Stay in touch
- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
- Website - [https://nestjs.com](https://nestjs.com/)
- Twitter - [@nestframework](https://twitter.com/nestframework)
## License
Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).