{"id":16874511,"url":"https://github.com/joway/node-kfk","last_synced_at":"2025-03-22T07:31:40.551Z","repository":{"id":31379421,"uuid":"127697064","full_name":"joway/node-kfk","owner":"joway","description":"A simplified kafka client based on node-rdkafka.","archived":false,"fork":false,"pushed_at":"2022-07-06T23:03:14.000Z","size":1143,"stargazers_count":19,"open_issues_count":16,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-14T15:32:58.181Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/joway/node-kfk","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joway.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-02T03:18:01.000Z","updated_at":"2024-04-20T19:34:07.000Z","dependencies_parsed_at":"2022-09-20T18:14:35.337Z","dependency_job_id":null,"html_url":"https://github.com/joway/node-kfk","commit_stats":null,"previous_names":["joway/node-rdkafka-promise"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joway%2Fnode-kfk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joway%2Fnode-kfk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joway%2Fnode-kfk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joway%2Fnode-kfk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joway","download_url":"https://codeload.github.com/joway/node-kfk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221824441,"owners_count":16886807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T15:33:04.962Z","updated_at":"2024-10-28T12:14:36.566Z","avatar_url":"https://github.com/joway.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Node-Kfk\n\n\u003cimg width=\"200px;\" src=\"logo.png\" /\u003e\n\n[![Build Status](https://travis-ci.org/joway/node-kfk.svg?branch=master)](https://travis-ci.org/joway/node-kfk)\n[![npm](https://img.shields.io/npm/v/kfk.svg)](https://www.npmjs.com/package/kfk)\n[![npm](https://img.shields.io/npm/dt/kfk.svg)](https://www.npmjs.com/package/kfk)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/51840be97e8d47d38fddb09cd95099ad)](https://app.codacy.com/app/joway/node-kfk?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=joway/node-kfk\u0026utm_campaign=Badge_Grade_Dashboard)\n[![Coverage Status](https://coveralls.io/repos/github/joway/node-kfk/badge.svg?branch=test-coverage)](https://coveralls.io/github/joway/node-kfk?branch=test-coverage)\n\n## Why I need it\n\nKafka is not friendly enough for programmers who don't have a clear knowledge on it.\n\nConsidering our usage are similar at most of the time, we want to provide a simple client for simple use case on kafka.\n\n## Usage\n\n### Install\n\n```shell\nnpm i kfk -S\n```\n\n### Kafka Producer\n\n```js\nconst conf = {\n  'client.id': 'kafka',\n  'metadata.broker.list': '127.0.0.1:9092',\n}\nconst topicConf = {\n}\nconst options = {\n  debug: false,\n}\n\nconst producer = new KafkaProducer(conf, topicConf, options)\n\nawait producer.connect()\n\nconsole.log('connected')\n\nwhile (true) {\n  const msg = `${new Date().getTime()}-${crypto.randomBytes(20).toString('hex')}`\n\n  await producer.produce(_.sample([\n    'rdkafka-test0',\n    'rdkafka-test1',\n    'rdkafka-test2',\n  ]), null, msg)\n}\n```\n\n### Kafka ALO(at least once) Consumer\n\n```js\nconst conf = {\n  'group.id': 'alo-consumer-test-1',\n  'metadata.broker.list': '127.0.0.1:9092',\n}\nconst topicConf = {\n  'auto.offset.reset': 'largest',\n}\nconst options = {\n  debug: false,\n}\n\nconst consumer = new KafkaALOConsumer(conf, topicConf, options)\nawait consumer.connect()\nawait consumer.subscribe([\n  'rdkafka-test0',\n  'rdkafka-test1',\n  'rdkafka-test2',\n])\n\nwhile (true) {\n  await consumer.consume(message =\u003e {\n    console.log(`topic: ${message.topic} offset : ${message.offset} val: ${message.value.toString('utf-8')}`)\n  }, {\n      size: 10,\n      concurrency: 5,\n    })\n}\n```\n\n### Kafka AMO(at most once) Consumer\n\n```js\nconst conf = {\n  'group.id': 'amo-consumer-test-1',\n  'metadata.broker.list': '127.0.0.1:9092',\n}\nconst topicConf = {\n  'auto.offset.reset': 'largest',\n}\nconst options = {\n  debug: false,\n}\n\nconst consumer = new KafkaAMOConsumer(conf, topicConf, options)\nawait consumer.connect()\nawait consumer.subscribe([\n  'rdkafka-test0',\n  'rdkafka-test1',\n  'rdkafka-test2',\n])\n\nwhile (true) {\n  await consumer.consume(message =\u003e {\n    console.log(`topic: ${message.topic} offset : ${message.offset} val: ${message.value.toString('utf-8')}`)\n  }, {\n      size: 10,\n      concurrency: 5,\n    })\n}\n```\n\n### Graceful Death\n\n```ts\nconst gracefulDeath = async () =\u003e {\n  await producer.die()\n  await consumer.die()\n  process.exit(0)\n}\nprocess.on('SIGINT', gracefulDeath)\nprocess.on('SIGQUIT', gracefulDeath)\nprocess.on('SIGTERM', gracefulDeath)\n```\n\n## Deep Dive\n\n### Choose your right consumer\n\n`node-kfk` provide two consumer choices for you: `KafkaALOConsumer` and `KafkaAMOConsumer`. `ALO` means `At Least Once`, and `AMO` means `At Most Once`.\n\n#### At Least Once\n\nIf you cannot tolerate any message loss and you have handled the repetitive execution situation in your consumer function, you may want your consumer has `at least once` guarantee.\n\n`KafkaALOConsumer` will monitor your consume callback function execute state and if there are any `Error` thrown in your consumer callback function (or process crashed), it will begin at the offsets you last consumed successfully.\n\n#### At Most Once\n\nIf you do not very care about little messages loss when problem happens, but you want to make sure that every message only can be handled on time, you can just use the `KafkaAMOConsumer`.\n\n`KafkaAMOConsumer` will auto commits the offsets when fetched the messages. It has better performance than `KafkaALOConsumer`, but not guarantee that all messages will be consumed.\n\n### Offset Management Detail\n\nIn `KafkaAMOConsumer`, `node-kfk` use the `enable.auto.commit=true` and `enable.auto.offset.store=true` options which completely depend on librdkafka to management the offsets and will auto commit the latest offsets periodically(the interval depends on `auto.commit.interval.ms`, default is `1000`).\n\nIn `KafkaALOConsumer`, we still want librdkafka to commit automatically, but we need to control offsetStore manually(now we set `enable.auto.commit=true` and `enable.auto.offset.store=false`). When `node-kfk` ensure that all messages had been handled successfully, it will store the latest offsets in offsetStore, and wait for committed by librdkafka.\n\n### Others\n\nThe client has been tested on:\n\n```yaml\n- os: linux\n  env: KAFKA_VERSION=0.10.2.2\n  node_js: 8\n- os: linux\n  env: KAFKA_VERSION=0.10.2.2\n  node_js: 10\n- os: linux\n  env: KAFKA_VERSION=0.11.0.3\n  node_js: 10\n- os: linux\n  env: KAFKA_VERSION=1.1.0\n  node_js: 10\n- os: linux\n  env: KAFKA_VERSION=2.0.0\n  node_js: 10\n```\n\nMore detailed document for `conf` and `topicConf` params in [librdkafka](https://github.com/edenhill/librdkafka) and [node-rdkafka](https://github.com/Blizzard/node-rdkafka)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoway%2Fnode-kfk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoway%2Fnode-kfk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoway%2Fnode-kfk/lists"}