{"id":32196224,"url":"https://github.com/clj-kafka/franzy","last_synced_at":"2025-10-22T02:29:23.682Z","repository":{"id":62431467,"uuid":"102736815","full_name":"clj-kafka/franzy","owner":"clj-kafka","description":"A set of Clojure libraries to work with Apache Kafka (producer, consumer, rebalancing, administration, and validation).","archived":false,"fork":false,"pushed_at":"2018-05-15T00:50:55.000Z","size":340,"stargazers_count":22,"open_issues_count":3,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-15T03:59:56.202Z","etag":null,"topics":["apache-kafka","clojure","kafka","kafka-client","kafka-consumer","kafka-producer"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clj-kafka.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-07T12:54:24.000Z","updated_at":"2024-09-06T12:50:44.000Z","dependencies_parsed_at":"2022-11-01T21:00:52.326Z","dependency_job_id":null,"html_url":"https://github.com/clj-kafka/franzy","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/clj-kafka/franzy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clj-kafka%2Ffranzy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clj-kafka%2Ffranzy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clj-kafka%2Ffranzy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clj-kafka%2Ffranzy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clj-kafka","download_url":"https://codeload.github.com/clj-kafka/franzy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clj-kafka%2Ffranzy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280366074,"owners_count":26318475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-22T02:00:06.515Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","clojure","kafka","kafka-client","kafka-consumer","kafka-producer"],"created_at":"2025-10-22T02:29:17.985Z","updated_at":"2025-10-22T02:29:23.665Z","avatar_url":"https://github.com/clj-kafka.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Franzy\n\nFranzy is a suite of Clojure libraries for [Apache Kafka](http://kafka.apache.org/documentation.html). It includes libraries for Kafka consumers, producers, partitioners, callbacks, serializers, and deserializers. \n\nAdditionally, there are libraries for administration, testing, mocking, running embedded Kafka brokers and zookeeper clusters, and more.\n\nThe main goal of Franzy is to make life easier for working with Kafka from Clojure. Franzy provides a foundation for building higher-level abstractions for whatever your needs might be.\n\n## Status\n\n[![Build Status (Develop)](https://travis-ci.org/clj-kafka/franzy.svg?branch=develop)](http://travis-ci.org/clj-kafka/franzy)\n\n\n## Platform\n\nFranzy breaks up its functionality into several different libraries to minimize dependency issues. Currently, the release and the develop branch target Kafka 0.11.0.0.\nAll the sub-projects are organized in this monorepository, but can be used (mostly) independently.\n\n\n### Core\n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/core.svg)](https://clojars.org/clj-kafka.franzy/core)\n\nClient library - core client-oriented functionality, i.e. consumer, producer, schemas, more.\n\n\n### Admin\n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/admin.svg)](https://clojars.org/clj-kafka.franzy/admin)\n\nAdminister Kafka with Clojure, get Clojure data in/out, create topics, add partitions, list brokers, etc.\n\n### Common\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/common.svg)](https://clojars.org/clj-kafka.franzy/common)\nCommon functionality for any Franzy development, and useful for Kafka in general\n\n\n### Nippy de/serializer      \n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/nippy.svg)](https://clojars.org/clj-kafka.franzy/nippy)\n\n[Nippy](https://github.com/ptaoussanis/nippy) Serializer/Deserializer for Kafka.\n\n### Transit de/serializer\n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/transit.svg)](https://clojars.org/clj-kafka.franzy/transit)\n\n[Transit](https://github.com/cognitect/transit-clj) Serializer/Deserializer for Kafka.\n\n\n### JSON de/serializer\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/json.svg)](https://clojars.org/clj-kafka.franzy/json)\n\nJSON/Smile Serializer/Deserializer for Kafka using [Cheshire](https://github.com/dakrone/cheshire).\n\n### Fressian de/serializer\n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/fressian.svg)](https://clojars.org/clj-kafka.franzy/fressian)\n\n[Fressian](https://github.com/clojure/data.fressian) Serializer/Deserializer for Kafka.\n\n### AVRO de/serializer\n\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/avro.svg)](https://clojars.org/clj-kafka.franzy/avro)\n\nAVRO Serializer/Deserializer for Kafka using [Abracad](https://github.com/damballa/abracad/blob/master/src/clojure/abracad/avro/edn.clj).\n\n### Embedded broker\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/embedded.svg)](https://clojars.org/clj-kafka.franzy/embedded)\n\nFull featured embedded Kafka server for testing/dev, with multiple implementations including concrete types and components.\n\n### Examples\n[![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/examples.svg)](https://clojars.org/clj-kafka.franzy/examples)\n\nGrowing project of examples using all the above, to learn at your leisure.\n\n\u003c!-- | [Franzy Mocks](https://github.com/ymilky/franzy-mocks)       | testing            | Test your consumers and producers without a running Kafka cluster, and more in the future.                                  | Franzy, Kafka client                                 | --\u003e\n\u003c!-- | [Travel Zoo](https://github.com/ymilky/travel-zoo)           | embedded Zookeeper | Embedded Zookeeper servers and clusters for testing and development, with concrete type and component versions available.   | Curator Test                                         |--\u003e\n\n\n## Features\n\n* Support for Kafka 0.11.0.0\n* Clojure types in and out of Kafka, no worrying about the Java API or Java types\n* Support for Clojure 1.8+\n* A light core of external dependencies to keep things light, future-proof, and in minimal conflict with your code\n* Comprehensive consumer API with support for both manual and automatic partition assignment as well as offset management\n* Producer API with support for synchronous and asynchronous production\n* Support for metadata and metrics\n* Choice of partitioning strategies (round-robin, range) and simple helpers/framework to implement your own\n* Validation for all significant data types, including configuration, via schema\n* Full, validated configuration from Clojure for Consumers, Producers, Brokers, and Kafka Connect - build your config as data\n* Protocols and conversions for implementing your own consumers, producers, tests, conversions, and more\n\u003c!-- * Mock producer and consumer, via [Franzy-Mocks](https://github.com/ymilky/franzy-mocks) --\u003e\n* Comprehensive Admin interface, including wrapping many undocumented/command-line only features via [![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/admin.svg)](https://clojars.org/clj-kafka.franzy/admin)\n* Helpers/framework for implementing custom callbacks for producers and consumers\n* Helpers/framework for implementing your own serializers/deserializers\n* Built-in serializers for keys and values for many data types and formats, including Strings, Integers, Longs, UUID, and Clojure Keywords, and EDN\n* Add-on serializers for Nippy, JSON/JSON SMILE, and Fressian, with more to come\n* A set of custom record types that fully wrap any data returned to and from Kafka, if you need, want, or prefer to use records rather than pure maps\n* Ability to pass any complex parameters using provided record types which also conform to validateable schemas\n* Embedded Kafka Server and components for testing via [![Clojars Project](https://img.shields.io/clojars/v/clj-kafka.franzy/embedded.svg)](https://clojars.org/clj-kafka.franzy/embedded)\n* Extensive examples, code comments, and documentation\n* More, coming soon....\n\n## Why?\n\nIn addition to raw features, some reasons you may want to use Franzy:\n\n* Comprehensive Kafka client\n* Extreme care to not remove, distort, break, or diminish anything in the existing Java API\n* Sane balance of performance vs. Clojure best-practices vs. ease-of-use\n* Does not force any viewpoint about producing, consuming, administration, etc. on you beyond what Kafka already does\n* À la carte - Lots of goodies and sugar, even for projects that are using mostly Java or don't need the consumer or producer at all. Build out what you need, no more, no less.\n* Currently being used in a real project, where Kafka is the \"spine\" of the application, and thus, must be updated, fixed, and changed as needed\n* Relatively future proof\n* Designed to be a good fit with stream processors, particularly [Onyx](https://github.com/onyx-platform/onyx)\n* See [Rationale](https://github.com/clj-kafka/franzy/blob/master/doc/rationale.md)\n\n## Requirements\n\nRequirements may vary slightly depending on your intended usage.\n\n* Clojure 1.8+ - You may be able to compile this library on/with earlier versions, but this is untested.\n* Kafka 0.11.0.0 - Some parts may work on earlier versions, but this is untested.\n\nA good way to get started with Kafka is to use Docker and/or Vagrant. I recommend using a Docker compose stack with Kafka and Zookeeper that lets you scale up/down to test. You can also use the embedded Kafka and Zookeeper libraries listed above and discussed in the Testing/Dev section.\n\n\n## Contributing/Roadmap\n\nThis library is still very young and is surely filled with bugs. Pull requests are welcome.\n\nThe following items are planned or have received partial development, I make no guarantees on timelines but plan to release some of these items in conjunction with getting other real-world work done using them:\n\n* Pool for holding on to consumers/producers and related objects where there is less of a clear path for managing the lifetime/instantiation of an object and disposing it. Some examples - Logging, Plugins for other libraries such as Onyx, Service Calls\n* Logging directly to Kafka via Timbre - dump logs directly into Kafka, presumably to process/dump them somewhere else. Useful for people with high log volumes or need some secondary processing of logs in a system like Logstash, Samza, Onyx, etc.\n* Some async helpers/patterns - Many of these might just be samples, but for more generic async tools, more may be released.\n* Additional tools and testing helpers, ex: parsing broker lists from franzy-admin directly to producer/consumer connnection strings.\n* Even more admin tools - combining some of the existing franzy-admin operations that are naturally part of common larger operations\n* Performance tweaks - some minor optimization and tweaks where possible given real-world benchmarking and usage\n* Additional partitioning strategies - ex: using different hashing techniques or supporting more narrow, but common use-cases for producer partitioning\n\nPlease contact me if any of these are high-demand for you so I can guage the urgency better.\n\nOf particular concern/value to fix/refactor/enhance currently:\n\n* Schemas - raw, and while working, may have mistakes, which in part may be to incorrect Kafka documentation. While the Kafka source was used for some of the harder parts of this library, much of the schema came from reading the documentation. Many mistakes have already been caught. Moreover, as Kafka grows and changes, config values are often the most in flux.\n* Serializers - More will be added as needed. Certainly the existing serializers can be enhanced and are meant as a blue-print and general usage. As your usage may vary, you may wish to fork or modify the existing serializers accordingly. In order to avoid this, options are easily injected into most serializers. Additional features can also be easily added by closing over values.\n* Testing - more unit tests are required, but adding them is an ongoing effort.\n* Field/Integration testing - If you're using this library in the wild, it may be too early, but I'd love to hear from you and fix any problems.\n* Async helpers, examples, particularly with core.async and/or manifold\n* Useful transformations/eductions/transductions/whatever for working with the various data structures to/from Kafka\n\nPlease be aware many problems/issues may be due to Kafka itself or the Java API. As such, before submitting issues, please check the Kafka official issue trackers first. If there is a reasonable workaround or solution, please leave a note and link to the underlying issues.\n\n## Contact\n\nFind me on [Clojurians Slack](https://clojurians.slack.com/) - @ymilky\n\n... or me on twitter [@chris_betz](https://www.twitter.com/chris_betz).\n\n## License\n\nCopyright © 2016 Yossi M. (ymilky).\n\nDistributed under the Eclipse Public License either version 1.0 or (at\nyour option) any later version.\n\nUse at your own risk, I am not responsible or liable. Please give credit if you use pieces of this library or otherwise, it is much appreciated.\n\n## Acknowledgements\n\nThanks to the following people for advice, support, code, and/or inspiration:\n\n* [Apache Kafka](http://kafka.apache.org) - Kafka Team, for the Java client, Kafka itself, docs, etc.\n* [Lucas Bradstreet](https://github.com/lbradstreet) \n* A thank you to all authors of other Kafka clients - for inspiration and creating valuable libraries I used until my requirements changed\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclj-kafka%2Ffranzy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclj-kafka%2Ffranzy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclj-kafka%2Ffranzy/lists"}