Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tawk/nestjs-batch-kafka
Process Kafka events by batch in NestJS
https://github.com/tawk/nestjs-batch-kafka
Last synced: about 9 hours ago
JSON representation
Process Kafka events by batch in NestJS
- Host: GitHub
- URL: https://github.com/tawk/nestjs-batch-kafka
- Owner: tawk
- License: mit
- Created: 2024-01-31T09:08:02.000Z (10 months ago)
- Default Branch: master
- Last Pushed: 2024-04-24T07:31:58.000Z (7 months ago)
- Last Synced: 2024-08-16T09:56:54.654Z (3 months ago)
- Language: TypeScript
- Size: 245 KB
- Stars: 1
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
[circleci-url]: https://circleci.com/gh/nestjs/nest
A progressive Node.js framework for building efficient and scalable server-side applications.
## Description
Process and Publish Kafka message by batch in NestJS. Cross compatible with `ServerKafka` and `ClientKafka` from `@nestjs/microservices` package.
## Installation
```bash
$ npm i --save @tawkto/nestjs-batch-kafka
```
## Overview
To use the batch kafka consumer, initialize `BatchKafkaServer` in your `main.ts` file by connecting the microservice to your app.
```typescript
const app = await NestFactory.createMicroservice(AppModule, {
// The config is the same as the KafkaOptions from the @nestjs/microservices package
strategy: new KafkaBatchServer({
client: {
brokers: ['localhost:52800', 'localhost:52801'],
},
consumer: {
groupId: 'test',
heartbeatInterval: 5000,
sessionTimeout: 30000,
},
run: {
autoCommitInterval: 5000,
autoCommitThreshold: 100,
partitionsConsumedConcurrently: 4,
},
})
})
```
Then you can start consuming the events in batches as follow
```typescript
@BatchProcessor('test')
async test(
@Payload() data: any[],
@Ctx() context: KafkaBatchContext,
) {
const heartbeat = context.getHeartbeat();
const resolveOffset = context.getResolveOffset();
const commitOffsetsIfNecessary = context.getCommitOffsetsIfNecessary();
await heartbeat();
for (const message of data) {
console.log(message);
}
resolveOffset(context.getMessages().at(-1).offset);
console.log("Batch resolved");
await heartbeat();
await commitOffsetsIfNecessary();
}
```
### Context
The `KafkaBatchContext` object provides the necessary components from `kafkajs`'s [`EachBatchPayload`](https://kafka.js.org/docs/consuming#a-name-each-batch-a-eachbatch):
Method
Type
Description
getMessages
KafkaMessage[]
Get the raw messages from Kafka in the batch
getConsumer
KafkaConsumer
Get the consumer instance
getResolveOffset
function
Get the resolve offset method
getHeartbeat
function
Get the heartbeat method
getPause
function
Get the pause method
getCommitOffsetsIfNecessary
function
Get the commit offsets if necessary method
getUncommittedOffsets
OffsetsByTopicPartition
Get the uncommitted offsets
getIsRunning
boolean
Indicate if the consumer is still running
getIsStale
boolean
Indicate if the consumer is stale
### Client
The `KafkaBatchClient` is exactly the same as the `KafkaClient` from the `@nestjs/microservices` package, except that `client.send` method is removed from the client as batch messages should not be used for `request-response` communication. On top of that, `KafkaBatchClient` also have the capability to [publish batch messages](https://kafka.js.org/docs/producing#producing-messages) or [publish to multiple topics](https://kafka.js.org/docs/producing#producing-to-multiple-topics) just like in `kafkajs`.
```typescript
@Module({
imports: [
ClientsModule.register([{
name: 'KAFKA_BATCH_CLIENT',
customClass: KafkaBatchClient,
options: {
client: {
brokers: ['localhost:52800', 'localhost:52801'],
},
consumer: {
groupId: 'test',
heartbeatInterval: 5000,
sessionTimeout: 30000,
},
},
}]),
],
})
export class AppModule {}
```
Then you can inject and use the `KafkaBatchClient` in your service as follow
```typescript
@Injectable()
export class AppService {
constructor(
@Inject('KAFKA_BATCH_CLIENT')
private kafkaClient: KafkaBatchClient,
) {}
async eventToBatch() {
this.kafkaClient.emit('test', { example: 'data'});
}
async publishBatch() {
// equivalent to kafkajs producer.send
this.kafkaClient.emitBatch('test', [{
example: 'data1'
}, {
example: 'data2'
}])
}
async publishBatchTopics() {
// will publish to two topics, topic1 and topic2
// equivalent to kafkajs producer.publishBatch
this.kafkaClient.emitBatchTopics([{
pattern: 'topic1',
data: [{ example: 'data11' }, { example: 'data12' }]
}, {
pattern: 'topic2',
data: [{ example: 'data21' }, { example: 'data22' }]
}])
}
}
```
Calling `send` with the `KafkaBatchClient` will result in an error.
```typescript
this.kafkaClient.send('send', { data: 'data'}); // Error
```