{"id":20224690,"url":"https://github.com/1337programming/kafka-node-wrapper","last_synced_at":"2026-05-05T14:05:08.552Z","repository":{"id":57288090,"uuid":"129973895","full_name":"1337programming/kafka-node-wrapper","owner":"1337programming","description":"kafka node wrapper with a sample docker setup","archived":false,"fork":false,"pushed_at":"2018-08-29T21:44:27.000Z","size":361,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-13T23:16:12.798Z","etag":null,"topics":["boilerplate","docker","docker-compose","kafka","kafka-consumer","kafka-docker-sample","kafka-producer","node","node-js","sample","wrapper"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/1337programming.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-17T22:50:39.000Z","updated_at":"2023-01-03T17:31:01.000Z","dependencies_parsed_at":"2022-09-11T07:23:53.957Z","dependency_job_id":null,"html_url":"https://github.com/1337programming/kafka-node-wrapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1337programming%2Fkafka-node-wrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1337programming%2Fkafka-node-wrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1337programming%2Fkafka-node-wrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1337programming%2Fkafka-node-wrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/1337programming","download_url":"https://codeload.github.com/1337programming/kafka-node-wrapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241663203,"owners_count":19999274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boilerplate","docker","docker-compose","kafka","kafka-consumer","kafka-docker-sample","kafka-producer","node","node-js","sample","wrapper"],"created_at":"2024-11-14T07:08:39.566Z","updated_at":"2025-10-17T13:22:13.395Z","avatar_url":"https://github.com/1337programming.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Docker Wrapper\n\n### Running\n\n* run `npm install`\n* run `docker-compose up`\n\nYou can now see the producer tab making new messages and the consumer tab reading those messages. You can start more producers (while still running 1 consumer) and see more messages coming in.\n\n### Topic Errors\n\nIf you are getting an error about the topic not existing, try running `docker-compose rm` to delete the containers. Then re-run `docker-compose up` again.\n\n## Use\n\nInstall\n`$ npm install kafka-node-wrapper --save`\n\n```javascript\nconst Kafka = require('kafka-node-wrapper'); // Node \u003c 9.x\n// import * as Kafka from 'kafka-node-wrapper' // Node \u003e 9.x\n\nconst consumer = new Kafka.Consumer();\nconst producer = new Kafka.Producer();\n\n// Connect with producer and consumer in parallel\nfunction connect() {\n  return Promise.all[consumer.connect(), producer.connect()];\n}\n\nfunction events() {\n  // Consumer Events\n  consumer.onMessage()\n    .subscribe((data) =\u003e {\n      console.log('Data', data);\n    });\n  consumer.onError()\n    .subscribe((err) =\u003e {\n      console.log('Consumer Error', err);\n    });\n\n  // Producer Events\n  producer.onReport()\n    .subscribe((report) =\u003e {\n      console.log('Producer Delivery Report', report);\n    });\n  producer.onError()\n    .subscribe((err) =\u003e {\n      console.log('Producer Error', err);\n    });\n\n}\n\n\n// DEMO!\nconnect()\n  .then(() =\u003e {\n    events();\n    return producer.publish(JSON.stringify({foo: 1, bar: 2}));\n  })\n  .catch((err) =\u003e {\n    console.error('Error!', err.message);\n  });\n```\n\n## Configuration\n\n### All\n\nConfigurations custom to this wrapper\n\n|   Field    | Description|   Type     | Default   |\n|------------|------------|------------|-----------|\n| throttle   | Throttle interval time (ms) | Number | 500|\n| topics     | Topics to subscribe to  | String[]  | \\['kafka-test-topic'\\] |\n| autoInterval | Allow auto intervals for polling (producer) and consuming (consumer). | boolean  | true |\n\n### Consumer\n\nConfigurations custom to this wrapper's Consumer class.\n\n|   Field    | Description|   Type     | Default   |\n|------------|------------|------------|-----------|\n| consumeMax   | Number of messages to consume for each interval. | Number | 1 |\n\nThis rest of the configuration is described [here](https://raw.githubusercontent.com/edenhill/librdkafka/0.11.1.x/CONFIGURATION.md).\n\n## Examples\n\n#### Basic Usage\n\nSample code for sample pub sub.\n```javascript\nconst Kafka = require('./src/index');\n\nconst consumer = new Kafka.consumer();\nconst producer = new Kafka.producer();\n\n/**\n * @param {Consumer} con\n * @return {Promise\u003cany\u003e} data\n */\nfunction consumerEvent(con) {\n  return new Promise((resolve, reject) =\u003e {\n    con.message()\n      .subscribe((data) =\u003e {\n        resolve(data);\n        console.log('SAMPLE Consumer Data:', data);\n      });\n    con.error()\n      .subscribe((err) =\u003e {\n        reject(err);\n        console.log('SAMPLE Consumer Error:', err);\n      });\n    console.log('SAMPLE Consumer Connected');\n    con.log()\n      .subscribe((log) =\u003e {\n        console.log('SAMPLE log', log);\n      });\n  });\n}\n\n/**\n * @param pro\n * @return {Promise\u003cany\u003e}\n */\nfunction producerEvent(pro) {\n  return new Promise((resolve, reject) =\u003e {\n    pro.report()\n      .subscribe((report) =\u003e {\n        console.log('SAMPLE Producer Delivery Report:', report);\n        return resolve(report);\n      });\n    pro.error()\n      .subscribe((err) =\u003e {\n        console.log('SAMPLE Producer Error:', err);\n        return reject(err);\n      });\n    console.log('SAMPLE Producer Connected');\n    const message = {\n      foo: 1,\n      bar: 2\n    };\n    pro.publish(JSON.stringify(message));\n  });\n}\n\nfunction main() {\n\n  Promise.all([consumer.connect(), producer.connect()])\n    .then(() =\u003e {\n      return Promise.all([consumerEvent(consumer), producerEvent(producer)]);\n    })\n    .then((data) =\u003e {\n      console.log('DATA', data);\n      return Promise.all([consumer.disconnect(), producer.disconnect()]);\n    })\n    .then(() =\u003e {\n      console.log('Disconnected');\n    })\n    .catch((err) =\u003e {\n      console.error('Error', err);\n    });\n\n}\n\nmain();\n\nmodule.exports = {\n  consumer: consumer,\n  producer: producer,\n  consumerEvent: consumerEvent,\n  producerEvent: producerEvent\n};\n\n```\n\n#### Overriding\n\nSample code for doing your own pulling or commit/consume\n\n```javascript\nconst KafkaWrapper = require('./src/index');\n\n/* ARBITRARY EXTERNAL CODE BEGIN */\n/**\n * This callback type is called `requestCallback` and is displayed as a global symbol.\n * @callback requestCallback\n * @param {{name: string, age: number}} responseCode\n */\n/**\n * Some stuff that has a callback (maybe write to a database or whatever)\n * @param data\n * @param {requestCallback} cb\n */\nfunction doStuffCB(data, cb) {\n  setTimeout(() =\u003e {\n    console.log('Did stuff to data');\n    return cb({name: 'Mary', age: 42});\n  }, 500);\n}\n\n/**\n * Promise Wrapper for our doStuffCB function\n * @param {Object} data - some data\n * @return {Promise\u003c{name: string, age: number}\u003e}\n */\nasync function cbWrapper(data) {\n  return new Promise((resolve, reject) =\u003e {\n    doStuffCB(data, (user) =\u003e {\n      return resolve(user);\n    });\n  });\n}\n\n/**\n * Does stuff\n * @param {Object} data - some data\n * @return {Promise\u003c{name: string, age: number}\u003e}\n */\nasync function doStuff(data) {\n  return new Promise((resolve, reject) =\u003e {\n    setTimeout(() =\u003e {\n      console.log('Did more stuff to data');\n      return resolve({name: 'Bob', age: 40});\n    }, 500);\n  });\n}\n/* ARBITRARY EXTERNAL CODE END */\n\nclass SampleOverride {\n\n  constructor() {\n    this._interval = null;\n    this.consumer = new KafkaWrapper.consumer({consumeMax: 10, autoInterval: false, topics: ['upload-user']}); // we must consume/commit ourselves\n    this.producer = new KafkaWrapper.producer({topics: ['upload-user']});\n  }\n\n  /**\n   * Run Service\n   * @param message\n   * @return {Promise\u003cvoid\u003e}\n   */\n  async run(message) {\n    // Connect\n    await this.consumer.connect();\n    await this.producer.connect();\n    return await this._scenario();\n  }\n\n  async _scenario() {\n    return new Promise(async (resolve, reject) =\u003e {\n      // Send message to publish something\n      await this.producer.publish(JSON.stringify({name: 'Dave', age: 38}));\n\n      // Start consuming 1 message every 500 ms\n      this._startConsuming(1);\n\n      // Listen to Messages\n      this.consumer.message().subscribe(async (message) =\u003e {\n        this._stopConsuming(); // Got a message stop consuming (this will not cancel other consumed messages)\n        await this._process(message); // Process Message\n        this._startConsuming(1); // Consume the next message\n      });\n\n      // Logging\n      this.consumer.error().subscribe((err) =\u003e {\n        console.error(err);\n      });\n\n      // End scenario based on some condition\n      setTimeout(async () =\u003e {\n        await this.producer.disconnect();\n        await this.consumer.disconnect();\n        return resolve();\n      }, 20000); // 20 seconds\n    });\n\n  }\n\n  /**\n   * Process Function\n   * @param message\n   * @return {Promise\u003cDeliveryReport\u003e}\n   */\n  async _process(message) {\n    const user1 = await cbWrapper(message); // Step 1\n    const user2 = await doStuff(message); // Step 2\n    const output = {\n      user1: user1,\n      user2: user2\n    };\n    return await this.producer.publish(JSON.stringify(output)); // Step 3 - Send\n  }\n\n  _startConsuming(numMessages) {\n    this._interval = setInterval(() =\u003e {\n      this.consumer.consume(numMessages);\n    }, 500);\n  }\n\n  _stopConsuming() {\n    clearInterval(this._interval);\n  }\n}\n\nconst override = new SampleOverride();\noverride.run({foo: 1, bar: 2})\n  .then(() =\u003e {\n    console.log('Done');\n    process.exit(0);\n  })\n  .catch((err) =\u003e {\n    console.error('Error', err);\n  });\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F1337programming%2Fkafka-node-wrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F1337programming%2Fkafka-node-wrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F1337programming%2Fkafka-node-wrapper/lists"}