{"id":19345933,"url":"https://github.com/tolitius/grete","last_synced_at":"2025-07-14T20:32:58.490Z","repository":{"id":56265763,"uuid":"218770402","full_name":"tolitius/grete","owner":"tolitius","description":"kafka client with threads and a scheduler","archived":false,"fork":false,"pushed_at":"2023-12-01T01:16:51.000Z","size":45,"stargazers_count":16,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-07-14T11:43:57.232Z","etag":null,"topics":["concurrency","consumer","kafka","producer","thread-pool"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tolitius.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-31T13:18:10.000Z","updated_at":"2024-03-13T10:31:02.000Z","dependencies_parsed_at":"2022-08-15T15:40:42.074Z","dependency_job_id":null,"html_url":"https://github.com/tolitius/grete","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tolitius/grete","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fgrete","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fgrete/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fgrete/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fgrete/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tolitius","download_url":"https://codeload.github.com/tolitius/grete/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fgrete/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265344832,"owners_count":23750566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","consumer","kafka","producer","thread-pool"],"created_at":"2024-11-10T04:08:23.132Z","updated_at":"2025-07-14T20:32:58.465Z","avatar_url":"https://github.com/tolitius.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# grete\n\nis [gregor](https://github.com/tolitius/grete/blob/master/src/grete/gregor.clj#L2)'s sister that adds a threadpool and a scheduler\n\n[![\u003c! release](https://img.shields.io/badge/dynamic/json.svg?label=release\u0026url=https%3A%2F%2Fclojars.org%2Ftolitius%2Fgrete%2Flatest-version.json\u0026query=version\u0026colorB=blue)](https://github.com/tolitius/grete/releases)\n[![\u003c! clojars\u003e](https://img.shields.io/clojars/v/tolitius/grete.svg)](https://clojars.org/tolitius/grete)\n\n... and some Java API\u003cbr/\u003e\n... and the latest kafka (at the moment of writing)\n\nthe idea behind `grete` is to be able to start a farm of kafka consumers that listen to (potentially) multiple topics and apply a simple consuming function.\n\n- [spilling the beans](#spilling-the-beans)\n  - [produce](#produce)\n  - [consume](#consume)\n  - [callbacks](#callbacks)\n- [Java API](#java-api)\n  - [several topics at once](#several-topics-at-once)\n- [License](#license)\n\n## spilling the beans\n\n```clojure\n$ make repl\n\n=\u003e (require '[grete.core :as g])\n```\n\nit is quite common for the same app to produce and consume,\u003cbr/\u003e\nhence we'll use one config for producing and consuming:\n\n```clojure\n=\u003e (def config {:kafka\n                {:producer\n                 {:bootstrap-servers \"1.1.1.1:9092,2.2.2.2:9092,3.3.3.3:9092\"}\n                 :consumer\n                 {:group-id \"foobar-consumer-group\"\n                  :bootstrap-servers \"1.1.1.1:9092,2.2.2.2:9092,3.3.3.3:9092\"\n                  :topics [\"foos\" \"bars\" \"bazs\"]\n                  :threads 42\n                  :poll-ms 100\n                  :auto-offset-reset \"earliest\"}}})\n```\n\n### produce\n\nproduce a couple of messages (to `foos` topic):\n\n```clojure\n=\u003e (def p (g/producer (get-in config [:kafka :producer])))\n\n;; send a couple of messages to topics: \"foos\" and \"bars\"\n=\u003e (g/send! p \"foos\" \"{:answer 42}\")\n=\u003e (g/send! p \"bars\" \"{:answer 42}\")\n\n=\u003e (g/close p)\n```\n\n### consume\n\na sample consuming function \"`process`\":\n\n```clojure\n;; the \"process\" function takes a batch of 'org.apache.kafka.clients.consumer.ConsumerRecords'\n;; which can be turned to a seq of maps with 'consumer-records-\u003emaps'\"\n\n=\u003e ;; not using \"consumer\" arg here, but you may\n   (defn process [consumer batch]\n     (let [batch (g/consumer-records-\u003emaps batch)\n           bsize (count batch)]\n       (when (pos? bsize)\n         (println \"picked up\" bsize \"events:\" batch))))\n```\n\nstart a farm of consumers (`42` threads as per config):\n\n```clojure\n=\u003e (def consumers (g/run-consumers process (get-in config [:kafka :consumer])))\n```\n\nonce the \"farm\" is started you'll see those two messages that were produces above:\n\n```clojure\n;;   picked up 2 events: ({:value {:answer 42},\n;;                         :key #object[[B 0x65ae581f [B@65ae581f],\n;;                         :partition 2,\n;;                         :topic foos,\n;;                         :offset 1000,\n;;                         :timestamp 1586888551200,\n;;                         :timestamp-type CreateTime}\n;;                        {:value {:answer 42},\n;;                         :key #object[[B 0x499b3437 [B@499b3437],\n;;                         :partition 13,\n;;                         :topic foos,\n;;                         :offset 3239,\n;;                         :timestamp 1586889147336,\n;;                         :timestamp-type CreateTime})\n```\n\nvalues here are strings, but could be byte arrays given bytearray de/serializers.\n\nas with other thread pools, it's a good idea to shut them down once we done working with them:\n\n```clojure\n=\u003e (g/stop-consumers consumers)\n```\n\n### callbacks\n\na kafka producer has an internal accumulator (kept in [Deque](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/Deque.html)) where it stores all the events before sending them out to the server (a.k.a. broker). when it is ready to send them, it splits the events stored in the accumulator in batches (controlled by a \"`batch.size`\" prop) and sends them out batch by batch.\n\nthe wait time before the actual \"publish | send\" is controlled via a \"`linger.ms`\" producer configuration property that maintains the balance of latency vs. throughput.\n\nhence the kafka publishing process is asynchronous by design.\n\nonce the events are published to the broker, kafka producer informs the calling API about the status via an optional _callback_.\n\nthis callback is a function that takes two arguments:\n\n* `metadata` in a form of\n```clojure\n{:offset 42\n :partition 13\n :topic \"eagle-nebula\"}\n```\n* and, in case of a problem, an `exception`\n\nthis callback can be provided to a grete's `send-then!` function:\n\n```clojure\n=\u003e (g/send-then! p \"foos\" \"{:answer 42}\"\n     (fn [metadata exception]\n       (println {:meta metadata\n                 :exception exception})))\n\n#object[org.apache.kafka.clients.producer.internals.FutureRecordMetadata 0x753e4eb5 \"org.apache.kafka.clients.producer.internals.FutureRecordMetadata@753e4eb5\"]\n{:meta {:offset 2\n        :partition 0\n        :topic foos}\n :exception nil}\n\n;; this part 👆 is returned to a producer via a callback\n;; this part 👇 is returned to a consumer\n\npicked up 1 events: ({:value {:answer 42}, :key nil, :partition 0, :topic foos, :offset 2, :timestamp 1701364230786, :timestamp-type CreateTime})\n```\n\n## Java API\n\nconsumer props:\n\n```yaml\nbootstrap-servers: \"1.1.1.1:9092,2.2.2.2:9092,3.3.3.3:9092\"\nthreads: 42\npoll-ms: 10\ntopics: \"foos,bars,bazs\"\ngroup-id: \"foobar-consumer-group\"\nauto-offset-reset: \"earliest\"\nenable-auto-commit: \"false\"\nheartbeat-interval-ms: \"3000\"\ndefault-api-timeout-ms: \"600000\"\nsession-timeout-ms: \"30000\"\n```\n\na mesage processing function:\n\n```java\nstatic void process(ConsumerRecords\u003cbyte[], byte[]\u003e records) {\n   // ...\n}\n```\n\na map of consumers:\n\n```java\nimport tolitius.Grete;\n\nBiConsumer\u003cKafkaConsumer, ConsumerRecords\u003cbyte[], byte[]\u003e\u003e consume =\n        (consumer, records) -\u003e process(records);\n\nMap consumers = Grete.startConsumers(consume, props);\n\nGrete.stopConsumers(consumers);\n```\n\ncould be \"`process(consumer, records)`\" if \"KafkaConsumer\" is also needed\n\n### several topics at once\n\nIn case the same group of consumer threads are listening to multiple topics _and_ the distinction needs to be made, i.e. what messages came from which topics, the records need to be groupped by topic:\n\n```java\nstatic Map\u003cString, List\u003cConsumerRecord\u003cbyte[], byte[]\u003e\u003e\u003e groupByTopic(ConsumerRecords\u003cbyte[], byte[]\u003e records) {\n\n    if (records.isEmpty()) {\n        log.trace(\"no new records in kafka, hence there is nothing to transport\");\n        return null;\n    }\n\n    var byTopic = new ConcurrentHashMap\u003cString, List\u003cConsumerRecord\u003cbyte[], byte[]\u003e\u003e\u003e();\n\n    records.forEach(record -\u003e {\n        var topic = record.topic();\n        var rs = byTopic.getOrDefault(topic, new ArrayList\u003c\u003e());\n        rs.add(record);\n        byTopic.put(record.topic(), rs);\n    });\n\n    return byTopic;\n}\n```\n\nthis is the \"`process`\" function from a previous example with a group by topic:\n\n```java\nstatic void process(ConsumerRecords\u003cbyte[], byte[]\u003e records) {\n\n    // since consumer may be subscribed to multiple topics the batch might include\n    // records of different types / from different topics.\n    // group all the the records in the batch by the topic to later pipe it to the proper function\n    var byTopic = groupByTopic(records);\n\n    if (byTopic != null) {\n        byTopic.forEach((topic, rs) -\u003e {\n\n            // ...\n        });\n    }\n}\n```\n\n## License\n\nCopyright © 2023 tolitius\n\nDistributed under the Eclipse Public License either version 1.0 or (at\nyour option) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolitius%2Fgrete","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftolitius%2Fgrete","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolitius%2Fgrete/lists"}