{"id":13692811,"url":"https://github.com/bobby/kafka-streams-clojure","last_synced_at":"2026-01-05T10:59:03.842Z","repository":{"id":138946313,"uuid":"91213549","full_name":"bobby/kafka-streams-clojure","owner":"bobby","description":"Clojure transducers interface to Kafka Streams","archived":false,"fork":false,"pushed_at":"2017-12-15T20:32:27.000Z","size":19,"stargazers_count":101,"open_issues_count":2,"forks_count":9,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-11-12T18:42:13.346Z","etag":null,"topics":["clojure","kafka","kafka-streams","transducers"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bobby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-05-14T01:55:05.000Z","updated_at":"2024-05-31T07:47:30.000Z","dependencies_parsed_at":"2024-04-08T02:16:53.080Z","dependency_job_id":"4a84a9fc-e511-4db8-ace7-7de61f2d5471","html_url":"https://github.com/bobby/kafka-streams-clojure","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobby%2Fkafka-streams-clojure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobby%2Fkafka-streams-clojure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobby%2Fkafka-streams-clojure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobby%2Fkafka-streams-clojure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bobby","download_url":"https://codeload.github.com/bobby/kafka-streams-clojure/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252095455,"owners_count":21693920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","kafka","kafka-streams","transducers"],"created_at":"2024-08-02T17:01:02.270Z","updated_at":"2026-01-05T10:59:03.835Z","avatar_url":"https://github.com/bobby.png","language":"Clojure","funding_links":[],"categories":["Kafka Streams"],"sub_categories":[],"readme":"# kafka-streams-clojure\n\n[Clojure transducers](https://clojure.org/reference/transducers)\ninterface to\n[Kafka Streams](https://kafka.apache.org/documentation/streams).  This\ncombo provides the best of both worlds for building streaming\napplications on Kafka with Clojure:\n\n* Simple, declarative, idiomatic, composable, testable stream\n  transformation logic, via transducers\n* Easy, battle-hardened distributed system topology specification,\n  cluster partition rebalancing, local state management, etc. via Kafka\n  Streams\n\n## Status\n\n**THIS LIBRARY IS CURRENTLY ALPHA STATUS, AND IS NOT FIT FOR PRODUCTION USE!**\n\nThis notice will be removed when I believe the API is stable and the\nlibrary has performed well under heavy loads in real-world use.\n\n### Features \u0026 Roadmap\n\nCurrently, this library supports:\n\n* Hooking a transducer into a `KStream` processing pipeline.\n\nIn the future, I plan for this library to support:\n\n* Helper transducers for stateful computations like joins, windowed\n  aggregates, etc. to mirror the functionality of the `KStream` API,\n  but which can be composed with purely functional steps\n* An appropriate level of integration into both the low-level\n  `Processor` API and the `KTable` APIs.\n\n## Installation\n\n**Note: Due to its alpha status, this library is not configured for\nCI/CD, and no JARs have been pushed to a public repository.  You'll\nhave to install (as per instructions below) into your local Maven repo\nbefore the following instructions will work**\n\nInclude the library JAR in your Boot/Leiningen dependencies:\n\n``` clojure\n[kafka-streams-clojure \"0.1.0-SNAPSHOT\"]\n```\n\n### Kafka Streams Dependency\n\nKafka Streams is included as a `provided` dependency, meaning your\napplication will need to include the\n[Kafka Streams JAR](https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams)\nas a dependency as well as this library.\n\n## Usage\n\nTransducers provide a more Clojure-idiomatic way to transform\nstreaming key value pairs than `KStream`'s Java 8 Streams-like API.\nThe key function is `kafka-streams-clojure.api/transduce-kstream`,\nwhich makes the given `KStream` a transducible context by applying the\ngiven transducer as a `Transformer`.  The step function is invoked\nwith the `ProcessorContext` and a 2-tuple of `[key value]` for each\nrecord, so the transducer should be shaped accordingly.\n\nThis library also provides a number of stateful transducers over Kafka\nStreams' Stores API for doing joins, windowed aggregates, etc.  The\ngoal of this library is to maintain feature parity with the high-level\n`KStream`, `KTable`, etc. APIs, as well as (eventually) to enable\ntransducer usage in the low-level `Processor` API.\n\n``` clojure\n// Start Kafka Cluster running locally\n\n(require '[kafka-streams-clojure.api :as api])\n(import '[org.apache.kafka.clients.producer KafkaProducer ProducerRecord]\n        '[org.apache.kafka.streams StreamsConfig KafkaStreams]\n        '[org.apache.kafka.streams.kstream KStreamBuilder])\n\n(def xform (comp (filter (fn [[k v]] (string? v)))\n                 (map (fn [[k v]] [v k]))\n                 (filter (fn [[k v]] (= \"foo\" v)))))\n(def builder (KStreamBuilder.))\n(def kstream (-\u003e builder\n                 (.stream (into-array String [\"tset\"]))\n                 (api/transduce-kstream xform)\n                 (.to \"test\")))\n\n(def kafka-streams\n  (KafkaStreams. builder\n                 (StreamsConfig. {StreamsConfig/APPLICATION_ID_CONFIG    \"test-app-id\"\n                                  StreamsConfig/BOOTSTRAP_SERVERS_CONFIG \"localhost:9092\"\n                                  StreamsConfig/KEY_SERDE_CLASS_CONFIG   org.apache.kafka.common.serialization.Serdes$StringSerde\n                                  StreamsConfig/VALUE_SERDE_CLASS_CONFIG org.apache.kafka.common.serialization.Serdes$StringSerde})))\n(.start kafka-streams)\n\n(def producer (KafkaProducer. {\"bootstrap.servers\" \"localhost:9092\"\n                               \"acks\"              \"all\"\n                               \"retries\"           \"0\"\n                               \"key.serializer\"    \"org.apache.kafka.common.serialization.StringSerializer\"\n                               \"value.serializer\"  \"org.apache.kafka.common.serialization.StringSerializer\"}))\n\n@(.send producer (ProducerRecord. \"tset\" \"foo\" \"bar\"))\n// Observe message come across topic \"test\" via kafka-console-consumer\n\n@(.send producer (ProducerRecord. \"tset\" \"baz\" \"quux\"))\n// Observe message does not come across topic \"test\" via kafka-console-consumer\n\n(.close producer)\n(.close kafka-streams)\n```\n\n## Dev, Build, Test\n\nThis project uses [Leiningen](https://leiningen.org/) for dev, test,\nand build workflow.\n\n### Run Tests\n\nThe test include an embedded, single-node Kafka/ZooKeeper cluster that\nruns on demand.\n\n``` bash\nlein test\n```\n\n### Run REPL\n\nTo run via the REPL, you'll need to fire up a Kafka Cluster.\n\n``` bash\nlein repl\n```\n\n### Build and Push JAR\n\n``` bash\nlein jar\nlein deploy\n```\n\n## License\n\n```\nCopyright 2017 Bobby Calderwood\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobby%2Fkafka-streams-clojure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbobby%2Fkafka-streams-clojure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobby%2Fkafka-streams-clojure/lists"}