{"id":26684243,"url":"https://github.com/johnapache/egg-kafka-node","last_synced_at":"2025-04-12T14:42:35.148Z","repository":{"id":34041214,"uuid":"166658534","full_name":"JohnApache/egg-kafka-node","owner":"JohnApache","description":"kafka plugin for egg.js","archived":false,"fork":false,"pushed_at":"2023-01-23T21:57:38.000Z","size":1631,"stargazers_count":38,"open_issues_count":20,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T09:19:20.765Z","etag":null,"topics":["egg","egg-kafka","egg-kafka-node","egg-plugin","kafka","kafka-node"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JohnApache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-20T12:27:06.000Z","updated_at":"2025-02-05T02:26:09.000Z","dependencies_parsed_at":"2023-02-13T03:45:16.105Z","dependency_job_id":null,"html_url":"https://github.com/JohnApache/egg-kafka-node","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnApache%2Fegg-kafka-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnApache%2Fegg-kafka-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnApache%2Fegg-kafka-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnApache%2Fegg-kafka-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JohnApache","download_url":"https://codeload.github.com/JohnApache/egg-kafka-node/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248583321,"owners_count":21128562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["egg","egg-kafka","egg-kafka-node","egg-plugin","kafka","kafka-node"],"created_at":"2025-03-26T09:19:26.060Z","updated_at":"2025-04-12T14:42:35.125Z","avatar_url":"https://github.com/JohnApache.png","language":"JavaScript","readme":"# egg-kafka-node\n\n[![NPM version][npm-image]][npm-url]\n[![build status][travis-image]][travis-url]\n[![Test coverage][codecov-image]][codecov-url]\n[![Known Vulnerabilities][snyk-image]][snyk-url]\n[![npm download][download-image]][download-url]\n\n[npm-image]: https://img.shields.io/npm/v/egg-kafka-node.svg?style=flat-square\n[npm-url]: https://npmjs.org/package/egg-kafka-node\n[travis-image]: https://www.travis-ci.org/JohnApache/events-proxy.svg\n[travis-url]: https://travis-ci.org/JohnApache/egg-kafka-node\n[codecov-image]: https://codecov.io/gh/JohnApache/egg-kafka-node/branch/master/graph/badge.svg\n[codecov-url]: https://codecov.io/gh/JohnApache/egg-kafka-node\n[snyk-image]: https://snyk.io/test/github/JohnApache/egg-kafka-node/badge.svg?targetFile=package.json\n[snyk-url]: https://snyk.io/test/github/JohnApache/egg-kafka-node?targetFile=package.json\n[download-image]: https://img.shields.io/npm/dm/egg-kafka-node.svg?style=flat-square\n[download-url]: https://npmjs.org/package/egg-kafka-node\n\n- [English](README.md)\n- [简体中文](README.zh_CN.md)\n\nThis plug-in is a package of [kafka-node] (https://github.com/SOHU-Co/kafka-node). It is an egg-style plug-in for easy use in the environment of egg.js. It also provides a detailed configuration of methods for sending Kafka messages. Refer to [https://github.com/SOHU-Co/kafka-node] (https://github.com/SOHU-Co/kafka-node). \n\n\n## Install\n\n```bash\n$ npm i egg-kafka-node --save\n```\n\n## Usage\n\n```js\n// {app_root}/config/plugin.js\nexports.kafkaNode = {\n  enable: true,\n  package: 'egg-kafka-node',\n};\n```\n\n## Configuration\n\n```js\n// {app_root}/config/config.default.js\nexports.kafkaNode = {\n  kafkaHost: '127.0.0.1:9092', // kafka connect host\n  clientOption: {}, // KafkaClient option, more documentation please visit kafka-node\n  consumerOption: [{\n    groupId: 'group1', // consumerGroup's groupId\n    topics: [ 'testTopic1' ], // topics under the same consumer group \n    options: {\n      fetchMaxWaitMs: 100,\n      fetchMinBytes: 1,\n      fetchMaxBytes: 1024 * 1024,\n    }, // relevant configuration for each consumer group, more documentation please visit kafka-node\n  }, {\n    groupId: 'group2',\n    topics: [ 'testTopic2' ],\n    options: {},\n  }, {\n    groupId: 'group3',\n    topics: [ 'testTopic3' ],\n  }],\n  \n  // HighLevelProducer option, more documentation please visit kafka-node\n  producerOption: {\n    requireAcks: 1, \n    ackTimeoutMs: 100, \n    partitionerType: 2, \n    autoCreateTopic: true, // Whether to turn on automatic topic creation. default true\n    topics: [ 'testTopic1', 'testTopic2', 'testTopic3' ], // Topics that all consumers need to consume\n  },\n  messageOption: {\n    partition: 0,\n    attributes: 0, // send message option\n  },\n  // NewConfig \n  baseConsumersDir: './app/kafka', // support read consumers files base dir\n};\n```\n\nsee [config/config.default.js](test/fixtures/apps/config/config.default.js) for more detail.\n\n## Structure\n```js\negg-project\n├── package.json\n├── app.js (optional)\n├── app\n|   ├── router.js\n│   ├── controller\n│   |   └── home.js\n│   ├── service (optional)\n│   |   └── user.js\n│   |   └── response_time.js\n│   └── kafka (optional)  --------\u003e like `controller, service...`\n│       ├── someTopic (optional)  -------\u003e topic name of kafka\n│            ├── someKey1Consumer.js(optional)  ------\u003e `someKey1` is a key of someTopic\n|            └── someKey2Consumer.js(optional)  ------\u003e `someKey2` is an another key of someTopic\n|            └── someKey3Consumer.ts(optional)  ------\u003e support load ts file consumers\n├── config\n|   ├── plugin.js\n|   ├── config.default.js\n│   ├── config.prod.js\n|   ├── config.test.js (optional)\n|   ├── config.local.js (optional)\n|   └── config.unittest.js (optional)\n\n```  \n## USE TIPS\n\n\u003e Note: The producer option topics of the kafkaNode configuration must create a corresponding topic directory under the {app-root}/kafka directory. Kafka-node automatically reads the file containing the'Consumers'filename under the topic, and the key needs to be passed in when sendMessage to facilitate business differentiation.\n\n\u003e Note: You must set app.config.baseDir, kafka need to load consumers base on the baseDir.\n\n\u003e Note: SendMessage messages max bytes depending on the configuration of you set.\n\n## Example\n\n```js\n// {app_root}/controller/index.js\nclass IndexController extends Controller {\n  async index() {\n    await this.ctx.kafka.sendMessage({\n      topic: 'someTopic', // Specify topics in the Kafka directory\n      key: 'someKey', // Specify consumer for the corresponding key under topic\n      messages: JSON.stringify({\n        username: 'JohnApache',\n        userId: 10001,\n        gender: 0\n      })\n    });\n  }\n\n  async some() {\n    this.ctx.kafka.sendMessageSync({\n      topic: 'someTopic', // Specify topics in the Kafka directory\n      key: 'someKey', // Specify consumer for the corresponding key under topic\n      messages: JSON.stringify({\n        username: 'JohnApache',\n        userId: 10001,\n        gender: 0\n      })\n    }, () =\u003e {\n      // success callback \n    }, () =\u003e {\n      // error callback \n    })\n  }\n}\n\n// {app_root}/kafka/someTopic/someKeyConsumer.js\nclass SomeKeySubscription extends Subscription {\n  async subscribe(message) {\n    const {value, topic, key} = message;\n    this.ctx.logger.info(`consume message ${value} by topic ${topic} key ${key} consumer`);\n    await asyncTask();\n  }\n}\n```\n\n## Questions \u0026 Suggestions\nPlease open an issue [here](https://github.com/JohnApache/egg-kafka-node/issues).\n\n## License\n\n[MIT](LICENSE)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnapache%2Fegg-kafka-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnapache%2Fegg-kafka-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnapache%2Fegg-kafka-node/lists"}