{"id":15032551,"url":"https://github.com/blizzard/node-rdkafka","last_synced_at":"2025-05-11T14:02:40.944Z","repository":{"id":38419314,"uuid":"65489497","full_name":"Blizzard/node-rdkafka","owner":"Blizzard","description":"Node.js bindings for librdkafka","archived":false,"fork":false,"pushed_at":"2025-05-02T18:58:31.000Z","size":11789,"stargazers_count":2157,"open_issues_count":116,"forks_count":402,"subscribers_count":58,"default_branch":"master","last_synced_at":"2025-05-11T14:01:45.488Z","etag":null,"topics":["kafka","librdkafka","nodejs"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Blizzard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-08-11T17:44:28.000Z","updated_at":"2025-05-09T06:56:05.000Z","dependencies_parsed_at":"2024-04-16T17:45:46.309Z","dependency_job_id":"0bb22a72-84d4-4562-b15b-bbd31ce66e4b","html_url":"https://github.com/Blizzard/node-rdkafka","commit_stats":{"total_commits":504,"total_committers":100,"mean_commits":5.04,"dds":0.6805555555555556,"last_synced_commit":"a950a56a43d679176209b43ed3e5817d0c1097ac"},"previous_names":[],"tags_count":70,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blizzard%2Fnode-rdkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blizzard%2Fnode-rdkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blizzard%2Fnode-rdkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blizzard%2Fnode-rdkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Blizzard","download_url":"https://codeload.github.com/Blizzard/node-rdkafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253576264,"owners_count":21930169,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","librdkafka","nodejs"],"created_at":"2024-09-24T20:18:42.679Z","updated_at":"2025-05-11T14:02:40.919Z","avatar_url":"https://github.com/Blizzard.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"node-rdkafka - Node.js wrapper for Kafka C/C++ library\n==============================================\n\nCopyright (c) 2016 Blizzard Entertainment.\n\n[https://github.com/blizzard/node-rdkafka](https://github.com/blizzard/node-rdkafka)\n\n[![Build Status](https://github.com/Blizzard/node-rdkafka/actions/workflows/test.yml/badge.svg)](https://github.com/Blizzard/node-rdkafka/actions/workflows/test.yml)\n\n[![npm version](https://badge.fury.io/js/node-rdkafka.svg)](https://badge.fury.io/js/node-rdkafka)\n\n# Looking for Collaborators!\n\nI am looking for *your* help to make this project even better! If you're interested, check [this out](https://github.com/Blizzard/node-rdkafka/issues/628)\n\n# Overview\n\nThe `node-rdkafka` library is a high-performance NodeJS client for [Apache Kafka](http://kafka.apache.org/) that wraps the native  [librdkafka](https://github.com/edenhill/librdkafka) library.  All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.\n\n__This library currently uses `librdkafka` version `2.10.0`.__\n\n## Reference Docs\n\nTo view the reference docs for the current version, go [here](https://blizzard.github.io/node-rdkafka/current/)\n\n## Contributing\n\nFor guidelines on contributing please see [CONTRIBUTING.md](https://github.com/blizzard/node-rdkafka/blob/master/CONTRIBUTING.md)\n\n## Code of Conduct\n\nPlay nice; Play fair.\n\n## Requirements\n\n* Apache Kafka \u003e=0.9\n* Node.js \u003e=16\n* Linux/Mac\n* Windows?! See below\n* OpenSSL\n\n### Mac OS High Sierra / Mojave\n\nOpenSSL has been upgraded in High Sierra and homebrew does not overwrite default system libraries. That means when building node-rdkafka, because you are using openssl, you need to tell the linker where to find it:\n\n```sh\nexport CPPFLAGS=-I/usr/local/opt/openssl/include\nexport LDFLAGS=-L/usr/local/opt/openssl/lib\n```\n\nThen you can run `npm install` on your application to get it to build correctly.\n\n__NOTE:__ From the `librdkafka` docs\n\n\u003e WARNING: Due to a bug in Apache Kafka 0.9.0.x, the ApiVersionRequest (as sent by the client when connecting to the broker) will be silently ignored by the broker causing the request to time out after 10 seconds. This causes client-broker connections to stall for 10 seconds during connection-setup before librdkafka falls back on the `broker.version.fallback` protocol features. The workaround is to explicitly configure `api.version.request` to `false` on clients communicating with \u003c=0.9.0.x brokers.\n\n### Alpine\n\nUsing Alpine Linux? Check out the [docs](https://github.com/Blizzard/node-rdkafka/blob/master/examples/docker-alpine.md).\n\n### Windows\n\nWindows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.2.10.0.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.\n\nRequirements:\n * [node-gyp for Windows](https://github.com/nodejs/node-gyp#on-windows)\n\n**Note:** I _still_ do not recommend using `node-rdkafka` in production on Windows. This feature was in high demand and is provided to help develop, but we do not test against Windows, and windows support may lag behind Linux/Mac support because those platforms are the ones used to develop this library. Contributors are welcome if any Windows issues are found :)\n\n## Tests\n\nThis project includes two types of unit tests in this project:\n* end-to-end integration tests\n* unit tests\n\nYou can run both types of tests by using `Makefile`. Doing so calls `mocha` in your locally installed `node_modules` directory.\n\n* Before you run the tests, be sure to init and update the submodules:\n  1. `git submodule init`\n  2. `git submodule update`\n* To run the unit tests, you can run `make lint` or `make test`.\n* To run the integration tests, you must have a running Kafka installation available. By default, the test tries to connect to `localhost:9092`; however, you can supply the `KAFKA_HOST` environment variable to override this default behavior. Run `make e2e`.\n\n# Usage\n\nYou can install the `node-rdkafka` module like any other module:\n\n```\nnpm install node-rdkafka\n```\n\nTo use the module, you must `require` it.\n\n```js\nconst Kafka = require('node-rdkafka');\n```\n\n## Configuration\n\nYou can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.10.0/CONFIGURATION.md)\n\nConfiguration keys that have the suffix `_cb` are designated as callbacks. Some\nof these keys are informational and you can choose to opt-in (for example, `dr_cb`). Others are callbacks designed to\nreturn a value, such as `partitioner_cb`.\n\nNot all of these options are supported.\nThe library will throw an error if the value you send in is invalid.\n\nThe library currently supports the following callbacks:\n* `partitioner_cb`\n* `dr_cb` or `dr_msg_cb`\n* `event_cb`\n* `rebalance_cb` (see [Rebalancing](#rebalancing))\n* `offset_commit_cb` (see [Commits](#commits))\n\n### Librdkafka Methods\n\nThis library includes two utility functions for detecting the status of your installation. Please try to include these when making issue reports where applicable.\n\nYou can get the features supported by your compile of `librdkafka` by reading the variable \"features\" on the root of the `node-rdkafka` object.\n\n```js\nconst Kafka = require('node-rdkafka');\nconsole.log(Kafka.features);\n\n// #=\u003e [ 'gzip', 'snappy', 'ssl', 'sasl', 'regex', 'lz4' ]\n```\n\nYou can also get the version of `librdkafka`\n\n```js\nconst Kafka = require('node-rdkafka');\nconsole.log(Kafka.librdkafkaVersion);\n\n// #=\u003e 2.10.0\n```\n\n## Sending Messages\n\nA `Producer` sends messages to Kafka.  The `Producer` constructor takes a configuration object, as shown in the following example:\n\n```js\nconst producer = new Kafka.Producer({\n  'metadata.broker.list': 'kafka-host1:9092,kafka-host2:9092'\n});\n```\n\nA `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.10.0/CONFIGURATION.md) file described previously.\n\nThe following example illustrates a list with several `librdkafka` options set.\n\n```js\nconst producer = new Kafka.Producer({\n  'client.id': 'kafka',\n  'metadata.broker.list': 'localhost:9092',\n  'compression.codec': 'gzip',\n  'retry.backoff.ms': 200,\n  'message.send.max.retries': 10,\n  'socket.keepalive.enable': true,\n  'queue.buffering.max.messages': 100000,\n  'queue.buffering.max.ms': 1000,\n  'batch.num.messages': 1000000,\n  'dr_cb': true\n});\n```\n\n#### Stream API\n\nYou can easily use the `Producer` as a writable stream immediately after creation (as shown in the following example):\n\n```js\n// Our producer with its Kafka brokers\n// This call returns a new writable stream to our topic 'topic-name'\nconst stream = Kafka.Producer.createWriteStream({\n  'metadata.broker.list': 'kafka-host1:9092,kafka-host2:9092'\n}, {}, {\n  topic: 'topic-name'\n});\n\n// Writes a message to the stream\nconst queuedSuccess = stream.write(Buffer.from('Awesome message'));\n\nif (queuedSuccess) {\n  console.log('We queued our message!');\n} else {\n  // Note that this only tells us if the stream's queue is full,\n  // it does NOT tell us if the message got to Kafka!  See below...\n  console.log('Too many messages in our queue already');\n}\n\n// NOTE: MAKE SURE TO LISTEN TO THIS IF YOU WANT THE STREAM TO BE DURABLE\n// Otherwise, any error will bubble up as an uncaught exception.\nstream.on('error', (err) =\u003e {\n  // Here's where we'll know if something went wrong sending to Kafka\n  console.error('Error in our kafka stream');\n  console.error(err);\n})\n```\n\nIf you do not want your code to crash when an error happens, ensure you have an `error` listener on the stream. Most errors are not necessarily fatal, but the ones that are will immediately destroy the stream. If you use `autoClose`, the stream will close itself at the first sign of a problem.\n\n#### Standard API\n\nThe Standard API is more performant, particularly when handling high volumes of messages.\nHowever, it requires more manual setup to use. The following example illustrates its use:\n\n```js\nconst producer = new Kafka.Producer({\n  'metadata.broker.list': 'localhost:9092',\n  'dr_cb': true\n});\n\n// Connect to the broker manually\nproducer.connect();\n\n// Wait for the ready event before proceeding\nproducer.on('ready', () =\u003e {\n  try {\n    producer.produce(\n      // Topic to send the message to\n      'topic',\n      // optionally we can manually specify a partition for the message\n      // this defaults to -1 - which will use librdkafka's default partitioner (consistent random for keyed messages, random for unkeyed messages)\n      null,\n      // Message to send. Must be a buffer\n      Buffer.from('Awesome message'),\n      // for keyed messages, we also specify the key - note that this field is optional\n      'Stormwind',\n      // you can send a timestamp here. If your broker version supports it,\n      // it will get added. Otherwise, we default to 0\n      Date.now(),\n      // you can send an opaque token here, which gets passed along\n      // to your delivery reports\n    );\n  } catch (err) {\n    console.error('A problem occurred when sending our message');\n    console.error(err);\n  }\n});\n\n// Any errors we encounter, including connection errors\nproducer.on('event.error', (err) =\u003e {\n  console.error('Error from producer');\n  console.error(err);\n})\n\n// We must either call .poll() manually after sending messages\n// or set the producer to poll on an interval (.setPollInterval).\n// Without this, we do not get delivery events and the queue\n// will eventually fill up.\nproducer.setPollInterval(100);\n```\n\nTo see the configuration options available to you, see the [Configuration](#configuration) section.\n\n##### Methods\n\n|Method|Description|\n|-------|----------|\n|`producer.connect()`| Connects to the broker. \u003cbr\u003e\u003cbr\u003e The `connect()` method emits the `ready` event when it connects successfully. If it does not, the error will be passed through the callback. |\n|`producer.disconnect()`| Disconnects from the broker. \u003cbr\u003e\u003cbr\u003eThe `disconnect()` method emits the `disconnected` event when it has disconnected. If it does not, the error will be passed through the callback. |\n|`producer.poll()` | Polls the producer for delivery reports or other events to be transmitted via the emitter. \u003cbr\u003e\u003cbr\u003eIn order to get the events in `librdkafka`'s queue to emit, you must call this regularly. |\n|`producer.setPollInterval(interval)` | Polls the producer on this interval, handling disconnections and reconnection. Set it to 0 to turn it off. |\n|`producer.produce(topic, partition, msg, key, timestamp, opaque)`| Sends a message. \u003cbr\u003e\u003cbr\u003eThe `produce()` method throws when produce would return an error. Ordinarily, this is just if the queue is full. |\n|`producer.flush(timeout, callback)`| Flush the librdkafka internal queue, sending all messages. Default timeout is 500ms |\n|`producer.initTransactions(timeout, callback)`| Initializes the transactional producer. |\n|`producer.beginTransaction(callback)`| Starts a new transaction. |\n|`producer.sendOffsetsToTransaction(offsets, consumer, timeout, callback)`| Sends consumed topic-partition-offsets to the broker, which will get committed along with the transaction. |\n|`producer.abortTransaction(timeout, callback)`| Aborts the ongoing transaction. |\n|`producer.commitTransaction(timeout, callback)`| Commits the ongoing transaction. |\n\n##### Events\n\nSome configuration properties that end in `_cb` indicate that an event should be generated for that option.  You can either:\n\n* provide a value of `true` and react to the event\n* provide a callback function directly\n\nThe following example illustrates an event:\n\n```js\nconst producer = new Kafka.Producer({\n  'client.id': 'my-client', // Specifies an identifier to use to help trace activity in Kafka\n  'metadata.broker.list': 'localhost:9092', // Connect to a Kafka instance on localhost\n  'dr_cb': true // Specifies that we want a delivery-report event to be generated\n});\n\n// Poll for events every 100 ms\nproducer.setPollInterval(100);\n\nproducer.on('delivery-report', (err, report) =\u003e {\n  // Report of delivery statistics here:\n  //\n  console.log(report);\n});\n```\n\nThe following table describes types of events.\n\n|Event|Description|\n|-------|----------|\n| `disconnected` | The `disconnected` event is emitted when the broker has disconnected. \u003cbr\u003e\u003cbr\u003eThis event is emitted only when `.disconnect` is called. The wrapper will always try to reconnect otherwise. |\n| `ready` | The `ready` event is emitted when the `Producer` is ready to send messages. |\n| `event` | The `event` event is emitted when `librdkafka` reports an event (if you opted in via the `event_cb` option). |\n| `event.log` | The `event.log` event is emitted when logging events come in (if you opted into logging via the `event_cb` option). \u003cbr\u003e\u003cbr\u003eYou will need to set a value for `debug` if you want to send information. |\n| `event.stats` | The  `event.stats` event is emitted when `librdkafka` reports stats (if you opted in by setting the `statistics.interval.ms` to a non-zero value). |\n| `event.error` | The  `event.error` event is emitted when `librdkafka` reports an error |\n| `event.throttle` | The `event.throttle` event emitted  when `librdkafka` reports throttling. |\n| `delivery-report` | The `delivery-report` event is emitted when a delivery report has been found via polling. \u003cbr\u003e\u003cbr\u003eTo use this event, you must set `request.required.acks` to `1` or `-1` in topic configuration and `dr_cb` (or `dr_msg_cb` if you want the report to contain the message payload) to `true` in the `Producer` constructor options. |\n\n### Higher Level Producer\n\nThe higher level producer is a variant of the producer which can propagate callbacks to you upon message delivery.\n\n```js\nconst producer = new Kafka.HighLevelProducer({\n  'metadata.broker.list': 'localhost:9092',\n});\n```\n\nThis will enrich the produce call so it will have a callback to tell you when the message has been delivered. You lose the ability to specify opaque tokens.\n\n```js\nproducer.produce(topicName, null, Buffer.from('alliance4ever'), null, Date.now(), (err, offset) =\u003e {\n  // The offset if our acknowledgement level allows us to receive delivery offsets\n  console.log(offset);\n});\n```\n\nAdditionally you can add serializers to modify the value of a produce for a key or value before it is sent over to Kafka.\n\n```js\nproducer.setValueSerializer((value) =\u003e {\n  return Buffer.from(JSON.stringify(value));\n});\n```\n\nOtherwise the behavior of the class should be exactly the same.\n\n## Kafka.KafkaConsumer\n\nTo read messages from Kafka, you use a `KafkaConsumer`.  You instantiate a `KafkaConsumer` object as follows:\n\n```js\nconst consumer = new Kafka.KafkaConsumer({\n  'group.id': 'kafka',\n  'metadata.broker.list': 'localhost:9092',\n}, {});\n```\n\nThe first parameter is the global config, while the second parameter is the topic config that gets applied to all subscribed topics. To view a list of all supported configuration properties, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) file described previously. Look for the `C` and `*` keys.\n\nThe `group.id` and `metadata.broker.list` properties are required for a consumer.\n\n### Rebalancing\n\nRebalancing is managed internally by `librdkafka` by default. If you would like to override this functionality, you may provide your own logic as a rebalance callback.\n\n```js\nconst consumer = new Kafka.KafkaConsumer({\n  'group.id': 'kafka',\n  'metadata.broker.list': 'localhost:9092',\n  'rebalance_cb': (err, assignment) =\u003e {\n\n    if (err.code === Kafka.CODES.ERRORS.ERR__ASSIGN_PARTITIONS) {\n      // Note: this can throw when you are disconnected. Take care and wrap it in\n      // a try catch if that matters to you\n      this.assign(assignment);\n    } else if (err.code == Kafka.CODES.ERRORS.ERR__REVOKE_PARTITIONS){\n      // Same as above\n      this.unassign();\n    } else {\n      // We had a real error\n      console.error(err);\n    }\n\n  }\n})\n```\n\n`this` is bound to the `KafkaConsumer` you have created. By specifying a `rebalance_cb` you can also listen to the `rebalance` event as an emitted event. This event is not emitted when using the internal `librdkafka` rebalancer.\n\n### Commits\n\nWhen you commit in `node-rdkafka`, the standard way is to queue the commit request up with the next `librdkafka` request to the broker. When doing this, there isn't a way to know the result of the commit. Luckily there is another callback you can listen to to get this information\n\n```js\nconst consumer = new Kafka.KafkaConsumer({\n  'group.id': 'kafka',\n  'metadata.broker.list': 'localhost:9092',\n  'offset_commit_cb': (err, topicPartitions) =\u003e {\n\n    if (err) {\n      // There was an error committing\n      console.error(err);\n    } else {\n      // Commit went through. Let's log the topic partitions\n      console.log(topicPartitions);\n    }\n\n  }\n})\n```\n\n`this` is bound to the `KafkaConsumer` you have created. By specifying an `offset_commit_cb` you can also listen to the `offset.commit` event as an emitted event. It receives an error and the list of topic partitions as argument. This is not emitted unless opted in.\n\n### Message Structure\n\nMessages that are returned by the `KafkaConsumer` have the following structure.\n\n```js\n{\n  value: Buffer.from('hi'), // message contents as a Buffer\n  size: 2, // size of the message, in bytes\n  topic: 'librdtesting-01', // topic the message comes from\n  offset: 1337, // offset the message was read from\n  partition: 1, // partition the message was on\n  key: 'someKey', // key of the message if present\n  timestamp: 1510325354780 // timestamp of message creation\n}\n```\n\n### Stream API\n\nThe stream API is the easiest way to consume messages. The following example illustrates the use of the stream API:\n\n```js\n// Read from the librdtesting-01 topic... note that this creates a new stream on each call!\nconst stream = KafkaConsumer.createReadStream(globalConfig, topicConfig, {\n  topics: ['librdtesting-01']\n});\n\nstream.on('data', (message) =\u003e {\n  console.log('Got message');\n  console.log(message.value.toString());\n});\n```\n\nYou can also get the `consumer` from the streamConsumer, for using consumer methods. The following example illustrates that:\n\n```js\nstream.consumer.commit(); // Commits all locally stored offsets\n```\n\n### Standard API\n\nYou can also use the Standard API and manage callbacks and events yourself.  You can choose different modes for consuming messages:\n\n* *Flowing mode*. This mode flows all of the messages it can read by maintaining an infinite loop in the event loop. It only stops when it detects the consumer has issued the `unsubscribe` or `disconnect` method.\n* *Non-flowing mode*. This mode reads a single message from Kafka at a time manually.\n\nThe following example illustrates flowing mode:\n```js\n// Flowing mode\nconsumer.connect();\n\nconsumer\n  .on('ready', () =\u003e {\n    consumer.subscribe(['librdtesting-01']);\n\n    // Consume from the librdtesting-01 topic. This is what determines\n    // the mode we are running in. By not specifying a callback (or specifying\n    // only a callback) we get messages as soon as they are available.\n    consumer.consume();\n  })\n  .on('data', (data) =\u003e {\n    // Output the actual message contents\n    console.log(data.value.toString());\n  });\n```\nThe following example illustrates non-flowing mode:\n```js\n// Non-flowing mode\nconsumer.connect();\n\nconsumer\n  .on('ready', () =\u003e {\n    // Subscribe to the librdtesting-01 topic\n    // This makes subsequent consumes read from that topic.\n    consumer.subscribe(['librdtesting-01']);\n\n    // Read one message every 1000 milliseconds\n    setInterval(() =\u003e {\n      consumer.consume(1);\n    }, 1000);\n  })\n  .on('data', (data) =\u003e {\n    console.log('Message found!  Contents below.');\n    console.log(data.value.toString());\n  });\n```\n\nThe following table lists important methods for this API.\n\n|Method|Description|\n|-------|----------|\n|`consumer.connect()` | Connects to the broker. \u003cbr\u003e\u003cbr\u003eThe `connect()` emits the event `ready` when it has successfully connected. If it does not, the error will be passed through the callback. |\n|`consumer.disconnect()` | Disconnects from the broker. \u003cbr\u003e\u003cbr\u003eThe `disconnect()` method emits `disconnected` when it has disconnected. If it does not, the error will be passed through the callback. |\n|`consumer.subscribe(topics)` | Subscribes to an array of topics. |\n|`consumer.unsubscribe()` | Unsubscribes from the currently subscribed topics. \u003cbr\u003e\u003cbr\u003eYou cannot subscribe to different topics without calling the `unsubscribe()` method first. |\n|`consumer.consume(cb)` | Gets messages from the existing subscription as quickly as possible. If `cb` is specified, invokes `cb(err, message)`. \u003cbr\u003e\u003cbr\u003eThis method keeps a background thread running to do the work. Note that the number of threads in nodejs process is limited by `UV_THREADPOOL_SIZE` (default value is 4) and using up all of them blocks other parts of the application that need threads. If you need multiple consumers then consider increasing `UV_THREADPOOL_SIZE` or using `consumer.consume(number, cb)` instead. |\n|`consumer.consume(number, cb)` | Gets `number` of messages from the existing subscription. If `cb` is specified, invokes `cb(err, message)`. |\n|`consumer.commit()` | Commits all locally stored offsets |\n|`consumer.commit(topicPartition)` | Commits offsets specified by the topic partition |\n|`consumer.commitMessage(message)` | Commits the offsets specified by the message |\n\nThe following table lists events for this API.\n\n|Event|Description|\n|-------|----------|\n|`data` | When using the Standard API consumed messages are emitted in this event. |\n|`partition.eof` | When using Standard API and the configuration option `enable.partition.eof` is set, `partition.eof` events are emitted in this event. The event contains `topic`, `partition` and `offset` properties. |\n|`warning` | The event is emitted in case of `UNKNOWN_TOPIC_OR_PART` or `TOPIC_AUTHORIZATION_FAILED` errors when consuming in *Flowing mode*. Since the consumer will continue working if the error is still happening, the warning event should reappear after the next metadata refresh. To control the metadata refresh rate set `topic.metadata.refresh.interval.ms` property. Once you resolve the error, you can manually call `getMetadata` to speed up consumer recovery. |\n|`rebalance` | The `rebalance` event is emitted when the consumer group is rebalanced. \u003cbr\u003e\u003cbr\u003eThis event is only emitted if the `rebalance_cb` configuration is set to a function or set to `true` |\n|`disconnected` | The `disconnected` event is emitted when the broker disconnects. \u003cbr\u003e\u003cbr\u003eThis event is only emitted when `.disconnect` is called. The wrapper will always try to reconnect otherwise. |\n|`ready` | The `ready` event is emitted when the `Consumer` is ready to read messages. |\n|`event` | The `event` event is emitted when `librdkafka` reports an event (if you opted in via the `event_cb` option).|\n|`event.log` | The `event.log` event is emitted when logging events occur (if you opted in for logging  via the `event_cb` option).\u003cbr\u003e\u003cbr\u003e You will need to set a value for `debug` if you want information to send. |\n|`event.stats` | The  `event.stats` event is emitted when `librdkafka` reports stats (if you opted in by setting the `statistics.interval.ms` to a non-zero value). |\n|`event.error` | The  `event.error` event is emitted when `librdkafka` reports an error |\n|`event.throttle` | The `event.throttle` event is emitted when `librdkafka` reports throttling.|\n\n## Reading current offsets from the broker for a topic\n\nSome times you find yourself in the situation where you need to know the latest (and earliest) offset for one of your topics. Connected producers and consumers both allow you to query for these through `queryWaterMarkOffsets` like follows:\n\n```js\nconst timeout = 5000, partition = 0;\nconsumer.queryWatermarkOffsets('my-topic', partition, timeout, (err, offsets) =\u003e {\n  const high = offsets.highOffset;\n  const low = offsets.lowOffset;\n});\n\nproducer.queryWatermarkOffsets('my-topic', partition, timeout, (err, offsets) =\u003e {\n  const high = offsets.highOffset;\n  const low = offsets.lowOffset;\n});\n\nAn error will be returned if the client was not connected or the request timed out within the specified interval.\n\n```\n\n## Metadata\n\nBoth `Kafka.Producer` and `Kafka.KafkaConsumer` include a `getMetadata` method to retrieve metadata from Kafka.\n\nGetting metadata on any connection returns the following data structure:\n\n```js\n{\n  orig_broker_id: 1,\n  orig_broker_name: \"broker_name\",\n  brokers: [\n    {\n      id: 1,\n      host: 'localhost',\n      port: 40\n    }\n  ],\n  topics: [\n    {\n      name: 'awesome-topic',\n      partitions: [\n        {\n          id: 1,\n          leader: 20,\n          replicas: [1, 2],\n          isrs: [1, 2]\n        }\n      ]\n    }\n  ]\n}\n```\n\nThe following example illustrates how to use the `getMetadata` method.\n\nWhen fetching metadata for a specific topic, if a topic reference does not exist, one is created using the default config.\nPlease see the documentation on `Client.getMetadata` if you want to set configuration parameters, e.g. `acks`, on a topic to produce messages to.\n\n```js\nconst opts = {\n  topic: 'librdtesting-01',\n  timeout: 10000\n};\n\nproducer.getMetadata(opts, (err, metadata) =\u003e {\n  if (err) {\n    console.error('Error getting metadata');\n    console.error(err);\n  } else {\n    console.log('Got metadata');\n    console.log(metadata);\n  }\n});\n```\n\n## Admin Client\n\n`node-rdkafka` now supports the admin client for creating, deleting, and scaling out topics. The `librdkafka` APIs also support altering configuration of topics and broker, but that is not currently implemented.\n\nTo create an Admin client, you can do as follows:\n\n```js\nconst Kafka = require('node-rdkafka');\n\nconst client = Kafka.AdminClient.create({\n  'client.id': 'kafka-admin',\n  'metadata.broker.list': 'broker01'\n});\n```\n\nThis will instantiate the `AdminClient`, which will allow the calling of the admin methods.\n\n```js\nclient.createTopic({\n  topic: topicName,\n  num_partitions: 1,\n  replication_factor: 1\n}, (err) =\u003e {\n  // Done!\n});\n```\n\nAll of the admin api methods can have an optional timeout as their penultimate parameter.\n\nThe following table lists important methods for this API.\n\n|Method|Description|\n|-------|----------|\n|`client.disconnect()` | Destroy the admin client, making it invalid for further use. |\n|`client.createTopic(topic, timeout, cb)` | Create a topic on the broker with the given configuration. See JS doc for more on structure of the topic object |\n|`client.deleteTopic(topicName, timeout, cb)` | Delete a topic of the given name |\n|`client.createPartitions(topicName, desiredPartitions, timeout, cb)` | Create partitions until the topic has the desired number of partitions. |\n\nCheck the tests for an example of how to use this API!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblizzard%2Fnode-rdkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblizzard%2Fnode-rdkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblizzard%2Fnode-rdkafka/lists"}