{"id":20471244,"url":"https://github.com/tawk/nestjs-batch-kafka","last_synced_at":"2025-03-05T13:32:39.805Z","repository":{"id":220251505,"uuid":"750759420","full_name":"tawk/nestjs-batch-kafka","owner":"tawk","description":"Process Kafka events by batch in NestJS","archived":false,"fork":false,"pushed_at":"2024-04-24T07:31:58.000Z","size":251,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-16T09:56:54.654Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tawk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-31T09:08:02.000Z","updated_at":"2024-04-24T07:31:54.000Z","dependencies_parsed_at":"2024-04-18T07:27:24.333Z","dependency_job_id":"a8fed2d0-4f52-4c2b-91b7-bbc9c717b241","html_url":"https://github.com/tawk/nestjs-batch-kafka","commit_stats":null,"previous_names":["tawk/nestjs-batch-kafka"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tawk%2Fnestjs-batch-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tawk%2Fnestjs-batch-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tawk%2Fnestjs-batch-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tawk%2Fnestjs-batch-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tawk","download_url":"https://codeload.github.com/tawk/nestjs-batch-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242034924,"owners_count":20061226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T14:15:32.572Z","updated_at":"2025-03-05T13:32:39.775Z","avatar_url":"https://github.com/tawk.png","language":"TypeScript","funding_links":["https://opencollective.com/nest","https://paypal.me/kamilmysliwiec"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"http://nestjs.com/\" target=\"blank\"\u003e\u003cimg src=\"https://nestjs.com/img/logo-small.svg\" width=\"200\" alt=\"Nest Logo\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456\n[circleci-url]: https://circleci.com/gh/nestjs/nest\n\n  \u003cp align=\"center\"\u003eA progressive \u003ca href=\"http://nodejs.org\" target=\"_blank\"\u003eNode.js\u003c/a\u003e framework for building efficient and scalable server-side applications.\u003c/p\u003e\n    \u003cp align=\"center\"\u003e\n\u003ca href=\"https://www.npmjs.com/~nestjscore\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/@nestjs/core.svg\" alt=\"NPM Version\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://www.npmjs.com/~nestjscore\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/npm/l/@nestjs/core.svg\" alt=\"Package License\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://www.npmjs.com/~nestjscore\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/npm/dm/@nestjs/common.svg\" alt=\"NPM Downloads\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://circleci.com/gh/nestjs/nest\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/circleci/build/github/nestjs/nest/master\" alt=\"CircleCI\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://coveralls.io/github/nestjs/nest?branch=master\" target=\"_blank\"\u003e\u003cimg src=\"https://coveralls.io/repos/github/nestjs/nest/badge.svg?branch=master#9\" alt=\"Coverage\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://discord.gg/G7Qnnhy\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/discord-online-brightgreen.svg\" alt=\"Discord\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/nest#backer\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/nest/backers/badge.svg\" alt=\"Backers on Open Collective\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/nest#sponsor\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/nest/sponsors/badge.svg\" alt=\"Sponsors on Open Collective\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://paypal.me/kamilmysliwiec\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/Donate-PayPal-ff3f59.svg\"/\u003e\u003c/a\u003e\n    \u003ca href=\"https://opencollective.com/nest#sponsor\"  target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg\" alt=\"Support us\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://twitter.com/nestframework\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/nestframework.svg?style=social\u0026label=Follow\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n  \u003c!--[![Backers on Open Collective](https://opencollective.com/nest/backers/badge.svg)](https://opencollective.com/nest#backer)\n  [![Sponsors on Open Collective](https://opencollective.com/nest/sponsors/badge.svg)](https://opencollective.com/nest#sponsor)--\u003e\n\n## Description\n\nProcess and Publish Kafka message by batch in NestJS. Cross compatible with `ServerKafka` and `ClientKafka` from `@nestjs/microservices` package.\n\n## Installation\n\n```bash\n$ npm i --save @tawkto/nestjs-batch-kafka\n```\n\n## Overview\nTo use the batch kafka consumer, initialize `BatchKafkaServer` in your `main.ts` file by connecting the microservice to your app.\n```typescript\nconst app = await NestFactory.createMicroservice\u003cMicroserviceOptions\u003e(AppModule, {\n\t// The config is the same as the KafkaOptions from the @nestjs/microservices package\n\tstrategy: new KafkaBatchServer({\n\t\tclient: {\n\t\t\tbrokers: ['localhost:52800', 'localhost:52801'],\n        },\n        consumer: {\n          groupId: 'test',\n          heartbeatInterval: 5000,\n          sessionTimeout: 30000,\n        },\n        run: {\n          autoCommitInterval: 5000,\n          autoCommitThreshold: 100,\n          partitionsConsumedConcurrently: 4,\n        },\n\t})\n})\n```\nThen you can start consuming the events in batches as follow\n```typescript\n@BatchProcessor('test')\n  async test(\n    @Payload() data: any[],\n    @Ctx() context: KafkaBatchContext,\n  ) {\n    const heartbeat = context.getHeartbeat();\n    const resolveOffset = context.getResolveOffset();\n    const commitOffsetsIfNecessary = context.getCommitOffsetsIfNecessary();\n\n    await heartbeat();\n\n    for (const message of data) {\n      console.log(message);\n    }\n\n    resolveOffset(context.getMessages().at(-1).offset);\n    console.log(\"Batch resolved\");\n\n    await heartbeat();\n    await commitOffsetsIfNecessary();\n  }\n```\n\n### Context\n\nThe `KafkaBatchContext` object provides the necessary components from `kafkajs`'s  [`EachBatchPayload`](https://kafka.js.org/docs/consuming#a-name-each-batch-a-eachbatch):\n\n\u003ctable\u003e\n\t\u003ctr\u003e\n\t\t\u003cth\u003eMethod\u003c/td\u003e\n\t\t\u003cth\u003eType\u003c/th\u003e\n\t\t\u003cth\u003eDescription\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetMessages\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003eKafkaMessage[]\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the raw messages from Kafka in the batch\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetConsumer\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003eKafkaConsumer\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the consumer instance\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetResolveOffset\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003efunction\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the resolve offset method\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetHeartbeat\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003efunction\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the heartbeat method\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetPause\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003efunction\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the pause method\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetCommitOffsetsIfNecessary\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003efunction\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the commit offsets if necessary method\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetUncommittedOffsets\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003eOffsetsByTopicPartition\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eGet the uncommitted offsets\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetIsRunning\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003eboolean\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eIndicate if the consumer is still running\u003c/td\u003e\n\t\u003c/tr\u003e\n\t\u003ctr\u003e\n\t\t\u003ctd\u003e\u003ccode\u003egetIsStale\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003e\u003ccode\u003eboolean\u003c/code\u003e\u003c/td\u003e\n\t\t\u003ctd\u003eIndicate if the consumer is stale\u003c/td\u003e\n\t\u003c/tr\u003e\n\u003c/table\u003e\n\n\n### Client\n\nThe `KafkaBatchClient` is exactly the same as the `KafkaClient` from the `@nestjs/microservices` package, except that `client.send` method is removed from the client as batch messages should not be used for `request-response` communication. On top of that, `KafkaBatchClient` also have the capability to [publish batch messages](https://kafka.js.org/docs/producing#producing-messages) or [publish to multiple topics](https://kafka.js.org/docs/producing#producing-to-multiple-topics) just like in `kafkajs`.\n\n```typescript\n@Module({\n\timports: [\n\t\tClientsModule.register([{\n\t\t\tname: 'KAFKA_BATCH_CLIENT',\n\t\t\tcustomClass: KafkaBatchClient,\n\t\t\toptions: {\n\t\t\t\tclient: {\n\t\t\t\tbrokers: ['localhost:52800', 'localhost:52801'],\n\t\t\t\t},\n\t\t\t\tconsumer: {\n\t\t\t\tgroupId: 'test',\n\t\t\t\theartbeatInterval: 5000,\n\t\t\t\tsessionTimeout: 30000,\n\t\t\t\t},\n\t\t\t},\n\t\t}]),\n\t],\n})\nexport class AppModule {}\n```\n\nThen you can inject and use the `KafkaBatchClient` in your service as follow\n```typescript\n@Injectable()\nexport class AppService {\n\tconstructor(\n\t\t@Inject('KAFKA_BATCH_CLIENT')\n\t\tprivate kafkaClient: KafkaBatchClient,\n\t) {}\n\n\tasync eventToBatch() {\n\t\tthis.kafkaClient.emit('test', { example: 'data'});\n\t}\n\n\tasync publishBatch() {\n\t\t// equivalent to kafkajs producer.send\n\t\tthis.kafkaClient.emitBatch('test', [{\n\t\t\texample: 'data1'\n\t\t}, {\n\t\t\texample: 'data2'\n\t\t}])\n\t}\n\n\tasync publishBatchTopics() {\n\t\t// will publish to two topics, topic1 and topic2\n\t\t// equivalent to kafkajs producer.publishBatch\n\t\tthis.kafkaClient.emitBatchTopics([{\n\t\t\tpattern: 'topic1',\n\t\t\tdata: [{ example: 'data11' }, { example: 'data12' }]\n\t\t}, {\n\t\t\tpattern: 'topic2',\n\t\t\tdata: [{ example: 'data21' }, { example: 'data22' }]\n\t\t}])\n\t}\n}\n```\n\nCalling `send` with the `KafkaBatchClient` will result in an error.\n```typescript\nthis.kafkaClient.send('send', { data: 'data'}); // Error\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftawk%2Fnestjs-batch-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftawk%2Fnestjs-batch-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftawk%2Fnestjs-batch-kafka/lists"}