{"id":13512500,"url":"https://github.com/nodefluent/kafka-streams","last_synced_at":"2025-05-15T16:02:50.469Z","repository":{"id":19611794,"uuid":"87479381","full_name":"nodefluent/kafka-streams","owner":"nodefluent","description":"equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:","archived":false,"fork":false,"pushed_at":"2023-03-05T09:37:45.000Z","size":4343,"stargazers_count":821,"open_issues_count":67,"forks_count":111,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-04-14T07:54:11.800Z","etag":null,"topics":["big-data","kafka","kafka-streams","node","nodejs","stream-processing","streams"],"latest_commit_sha":null,"homepage":"https://nodefluent.github.io/kafka-streams/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nodefluent.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-04-06T22:06:51.000Z","updated_at":"2024-04-09T21:30:42.000Z","dependencies_parsed_at":"2024-01-07T02:24:12.254Z","dependency_job_id":"a77af8bd-d172-4db2-89a5-442edfae001f","html_url":"https://github.com/nodefluent/kafka-streams","commit_stats":{"total_commits":236,"total_committers":21,"mean_commits":"11.238095238095237","dds":0.6440677966101696,"last_synced_commit":"91bf75cb0dd551739bfe9949625aa58ea5eefbd6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodefluent%2Fkafka-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodefluent%2Fkafka-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodefluent%2Fkafka-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodefluent%2Fkafka-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nodefluent","download_url":"https://codeload.github.com/nodefluent/kafka-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247737770,"owners_count":20987718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","kafka","kafka-streams","node","nodejs","stream-processing","streams"],"created_at":"2024-08-01T03:01:56.943Z","updated_at":"2025-04-07T22:05:40.343Z","avatar_url":"https://github.com/nodefluent.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","Development"],"sub_categories":["Client libraries"],"readme":"# node-kafka-streams\n\n[![Build Status](https://travis-ci.org/nodefluent/kafka-streams.svg?branch=master)](https://travis-ci.org/nodefluent/kafka-streams)\n[![npm version](https://badge.fury.io/js/kafka-streams.svg)](https://badge.fury.io/js/kafka-streams)\n\n```\n// suggested Node.js version: v12.16.1\nnpm install --save kafka-streams\n```\n\n```javascript\nconst {KafkaStreams} = require(\"kafka-streams\");\n\nconst config = require(\"./config.json\");\nconst factory = new KafkaStreams(config);\n\nconst kstream = factory.getKStream(\"input-topic\");\nconst ktable = factory.getKTable(/* .. */);\n\nkstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to(\"output-topic\");\n```\n\n\u003e CHANGES: The latest version brings a lot of changes, please check [here](CHANGELOG.md) before updating.\n\n## API Overview\n\n* [Quick Start](https://nodefluent.github.io/kafka-streams/docs/)\n* [Message 'to' and 'from' Apache Kafka](https://nodefluent.github.io/kafka-streams/docs/handlingMessageSchemas)\n* [API Info](https://nodefluent.github.io/kafka-streams/docs/classDoc)\n* [Documentation](https://nodefluent.github.io/kafka-streams/)\n* [Operator descriptions](https://nodefluent.github.io/kafka-streams/docs/mostAPI)\n* [Examples](https://github.com/nodefluent/kafka-streams/tree/master/examples)\n* [Native Client](https://nodefluent.github.io/kafka-streams/docs/native) | [SSL, SASL, Kerberos](https://nodefluent.github.io/kafka-streams/docs/sslSasl)\n\n## You might also like\n\n* [node-kafka-connect](https://github.com/nodefluent/kafka-connect)\n* [node-schema-registry](https://github.com/nodefluent/schema-registry)\n* [node-kafka-rest-ui](https://github.com/nodefluent/kafka-rest-ui)\n\n## README Overview\n\n- [node-kafka-streams](#node-kafka-streams)\n  - [API Overview](#api-overview)\n  - [You might also like](#you-might-also-like)\n  - [README Overview](#readme-overview)\n  - [Prerequisites](#prerequisites)\n    - [A note on native mode](#a-note-on-native-mode)\n  - [Aim of this Library](#aim-of-this-library)\n  - [Description](#description)\n  - [Port Progress Overview](#port-progress-overview)\n  - [Operator Implementations](#operator-implementations)\n  - [Additional Operators](#additional-operators)\n  - [Stream Action Implementations](#stream-action-implementations)\n  - [Join Operations](#join-operations)\n    - [KStream Status](#kstream-status)\n    - [KTable Status](#ktable-status)\n    - [KTable \u003c-\u003e KStream Status](#ktable---kstream-status)\n  - [Window Operations](#window-operations)\n    - [KStream](#kstream)\n- [More](#more)\n  - [Can I use this library yet?](#can-i-use-this-library-yet)\n  - [Are we ready for production yet?](#are-we-ready-for-production-yet)\n  - [Even More](#even-more)\n\n## Prerequisites\n\n- Kafka broker should be version `\u003e= 0.11.x`\n- Node.js should be version `\u003e= 8.x.x`\n\n### A note on native mode\n\nIf you are using the native mode (`config: { noptions: {} }`).\nYou will have to manually install `node-rdkafka` alongside kafka-streams.\n(This requires a Node.js version between 9 and 12 and will not work with Node.js \u003e= 13, last tested with 12.16.1)\n\nOn Mac OS High Sierra / Mojave:\n`CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4`\n\nOtherwise:\n`yarn add --frozen-lockfile node-rdkafka@2.7.4`\n\n(Please also note: Doing this with npm does not work, it will remove your deps, `npm i -g yarn`)\n\n## Aim of this Library\n\n- this is not a 1:1 port of the official JAVA kafka-streams\n- the goal of this project is to give at least the same options to\na nodejs developer that kafka-streams provides for JVM developers\n- stream-state processing, table representation, joins, aggregate etc.\nI am aiming for the easiest api access possible checkout the [word count example](https://github.com/nodefluent/kafka-streams/blob/master/examples/wordCount.js)\n\n## Description\n\n[kafka-streams](http://docs.confluent.io/3.0.0/streams) :octopus: equivalent for nodejs :sparkles::turtle::rocket::sparkles:\nbuild on super fast :fire: observables using [most.js](https://github.com/cujojs/most) :metal:\n\nships with [sinek](https://github.com/nodefluent/node-sinek) :pray: for backpressure\n\ncomes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features\n\nthe lib also comes with a few `window` operations that are more similar to [Apache Flink](https://flink.apache.org/),\nyet they still feel natural in this api :squirrel:\n\noverwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..\n\nasync (Promises) and sync stream operators e.g. `stream$.map()` or `stream$.asyncMap()`\n\nsuper easy API :goberserk:\n\nthe lib is based on `sinek`, which is based on kafka-node's `ConsumerGroups`\n\n## Port Progress Overview\n\n- [x] core structure\n- [x] KStream base - stream as a changelog\n- [x] KTable base - stream as a database\n- [x] KStream \u0026 KTable cloning\n- [x] complex stream join structure\n- [ ] advanced joins [see](#join-operations)\n- [ ] windows (for joins) [see](#join-operations)\n- [ ] flink like window operations\n- [x] word-count example\n- [x] more examples\n- [x] local-storage for etl actions\n- [x] local-storage factory (one per action)\n- [ ] KStorage example for any DB that supports atomic actions\n- [ ] backing-up local-storage via kafka\n- [x] kafka client implementation\n- [x] KTable replay to Kafka (produce)\n- [x] stream for topic message production only\n- [x] sinek implementation\n- [x] backpressure mode for KafkaClient\n- [x] auto-json payloads (read-map/write-map)\n- [x] auto producer partition and keyed-message handling\n- [x] documentation\n- [x] API description\n- [ ] higher join \u0026 combine examples\n- [x] embed native client `librdkafka` for more performance\n- [x] SSL\n- [x] SASL\n- [x] Kerberos\n\n\n## Operator Implementations\n\n- [x] map\n- [x] asyncMap\n- [x] constant\n- [x] scan\n- [x] timestamp\n- [x] tap\n- [x] filter\n- [x] skipRepeats\n- [x] skipRepeatsWith\n- [x] slice\n- [x] take\n- [x] skip\n- [x] takeWhile\n- [x] skipWhile\n- [x] until\n- [x] since\n- [x] reduce\n- [x] chainReduce\n- [x] forEach (observe)\n- [x] chainForEach\n- [x] drain\n- [x] _zip\n- [x] _merge\n- [x] _join\n- [x] _combine\n- [x] _sample\n- [x] throttle\n- [x] debounce\n- [x] delay\n- [x] multicast\n- A description of the operators can be found [here](docs/most-api.md)\n- Missing an operator? Feel free to open an issue :cop:\n\n## Additional Operators\n\n- [x] mapStringToArray\n- [x] mapArrayToKV\n- [x] mapStringToKV\n- [x] mapParse\n- [x] mapStringify\n- [x] atThroughput\n- [x] mapWrapKafkaPayload\n- [x] mapToFormat\n- [x] mapFromFormat\n- Want more? Feel free to open an issue :cop:\n\n## Stream Action Implementations\n\n- [x] countByKey\n- [x] sumByKey\n- [x] min\n- [x] max\n- Want more? Feel free to open an issue :cop:\n\n## Join Operations\n\n[Operation description](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics)\n\n### KStream Status\n\n- [x] merge\n- [ ] outerJoin\n- [x] innerJoin\n- [ ] leftJoin\n- [x] branch\n\n### KTable Status\n\n- [x] merge\n- [ ] outerJoin\n- [ ] innerJoin\n- [ ] leftJoin\n\n### KTable \u003c-\u003e KStream Status\n\n- [x] merge\n- [ ] outerJoin\n- [ ] innerJoin\n- [ ] leftJoin\n\n## Window Operations\n\n### KStream\n\n- [x] window\n- [ ] advanced window\n- [ ] rolling window\n\n# More\n\n## Can I use this library yet?\n\nYes.\n\n## Are we ready for production yet?\n\nProbably, yes. :smile:\n\n## Even More\n\nForks or Stars give motivation :bowtie:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodefluent%2Fkafka-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnodefluent%2Fkafka-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodefluent%2Fkafka-streams/lists"}