{"id":13476773,"url":"https://github.com/dharmeshkakadia/awesome-kafka","last_synced_at":"2026-02-05T20:36:17.042Z","repository":{"id":48499786,"uuid":"87770687","full_name":"dharmeshkakadia/awesome-kafka","owner":"dharmeshkakadia","description":"Everything about Apache Kafka","archived":false,"fork":false,"pushed_at":"2023-07-18T16:53:36.000Z","size":75,"stargazers_count":199,"open_issues_count":0,"forks_count":30,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-05-19T21:20:05.074Z","etag":null,"topics":["awesome","awsome-list","kafka"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dharmeshkakadia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":["dharmeshkakadia"]}},"created_at":"2017-04-10T05:38:47.000Z","updated_at":"2024-04-02T17:42:22.000Z","dependencies_parsed_at":"2024-01-12T04:44:20.138Z","dependency_job_id":"2f2443b8-15ce-44eb-95a0-cd103eca3a8f","html_url":"https://github.com/dharmeshkakadia/awesome-kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dharmeshkakadia%2Fawesome-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dharmeshkakadia%2Fawesome-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dharmeshkakadia%2Fawesome-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dharmeshkakadia%2Fawesome-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dharmeshkakadia","download_url":"https://codeload.github.com/dharmeshkakadia/awesome-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245784794,"owners_count":20671618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome","awsome-list","kafka"],"created_at":"2024-07-31T16:01:34.393Z","updated_at":"2026-02-05T20:36:17.000Z","avatar_url":"https://github.com/dharmeshkakadia.png","language":null,"funding_links":["https://github.com/sponsors/dharmeshkakadia"],"categories":["Others","Libraries","Other Lists","Misc"],"sub_categories":["Other Awesome Kafka","TeX Lists"],"readme":"# awesome-kafka\nEverything about Apache Kafka\n\nShare your Kafka ❤️ through pull requests :)\n\nWhat can you expect to see here?\n\n* [Development](#development)\n  * [Client libraries](#client-libraries)\n  * [Kafka Streams libraries](#kafka-streams-libraries)\n  * [KSQL](#ksql)\n  * [Connectors](#connectors)\n  * [Producers](#producers)\n  * [Consumers](#consumers)\n  * [Transformations](#transformations)\n  * [Testing](#testing)\n* [Operations](#operations)\n  * [CI/CD](#cicd)\n  * [Operational Utilities](#operational-utilities)\n  * [Monitoring](#monitoring)\n  * [Performance tools](#performance-tools)\n  * [Security](#security)\n  * [Audit](#audit)\n  * [Mirroring](#mirroring)\n  * [Backup](#backup)\n  * [Tools](#tools)\n  * [Metadata Management](#metadata-management)\n  * [Schema Management](#schema-management)\n  * [UI](#ui)\n* [Deployment](#deployment)\n  * [Kubernetes](#kubernetes)\n  * [Docker](#docker)\n  * [Integrations](#integrations)\n  * [Alternative Implementations](#alternative-implementations)\n* [Project built on top of Kafka](#projects-built-on-top-of-Kafka)\n* [Unsorted](#unsorted)\n\n\n## Development\n\n### Client libraries\n\n* [Go](https://github.com/Shopify/sarama)\n* [Go](https://github.com/jdamick/kafka)\n* [Go](https://github.com/optiopay/kafka)\n* [Go Kafka streaming](https://github.com/lovoo/goka)\n* [Node](https://kafka.js.org)\n* [Node](https://github.com/oleksiyk/kafka)\n* [Node](https://github.com/nodefluent/node-sinek)\n* [Node Kafka streams](https://github.com/nodefluent/kafka-streams)\n* [Node librdkafka](https://github.com/nodefluent/node-rdkafka)\n* [Php](https://github.com/weiboad/kafka-php)\n* [Php](https://github.com/arnaud-lb/php-rdkafka)\n* [Perl](https://github.com/TrackingSoft/Kafka)\n* [Python (asyncio)](https://github.com/aio-libs/aiokafka)\n* [Python (kafka-python)](https://github.com/dpkp/kafka-python)\n* [Python (pykafka)](https://github.com/Parsely/pykafka)\n* [Ruby](https://github.com/zendesk/ruby-kafka)\n* [Phobos - Ruby App framework](https://github.com/klarna/phobos)\n* [Karafka - Ruby and Rails framework](https://github.com/karafka/karafka)\n* [Dart](https://github.com/dart-kafka/kafka)\n* [JRuby](https://github.com/joekiller/jruby-kafka)\n* [Elixir](https://github.com/kafkaex/kafka_ex)\n* [Erlang](https://github.com/klarna/brod)\n* [Rust](https://github.com/spicavigo/kafka-rust)\n* [LinkedIn wrapper library](https://github.com/linkedin/li-apache-kafka-clients)\n* [kasper - a lightweight library for processing Kafka topics](https://github.com/movio/kasper)\n* and [others](https://cwiki.apache.org/confluence/display/KAFKA/Clients)\n\n### Kafka Streams Libraries\n* [Scala](https://github.com/lightbend/kafka-streams-scala)\n* [faust - Python Stream Processing](https://github.com/robinhood/faust)\n* [Southpaw - Streaming left joins for data change capture](https://github.com/jwplayer/southpaw)\n\n### KSQL\n* [KSQL JDBC Driver](https://github.com/mmolimar/ksql-jdbc-driver)\n* [KSQL Addon - User Defined Function (UDF) for Machine Learning](https://github.com/kaiwaehner/ksql-machine-learning-udf)\n\n### Connectors\n\n* [JDBC](https://github.com/confluentinc/kafka-connect-jdbc)\n* [Elasticsearch](https://github.com/confluentinc/kafka-connect-elasticsearch)\n* [HDFS](https://github.com/confluentinc/kafka-connect-hdfs)\n* [Beast - Kafka to BigQuery Sink](https://github.com/gojek/beast)\n* [BigQuery](https://github.com/nodefluent/bigquery-kafka-connect)\n* [SQL](https://github.com/nodefluent/sequelize-kafka-connect)\n* [GCloud pubsub](https://github.com/nodefluent/gcloud-pubsub-kafka-connect)\n* [BigTable](https://github.com/nodefluent/bigtable-kafka-connect)  \n* [Google Cloud Storage (GCS)](https://github.com/aiven/aiven-kafka-connect-gcs)\n* [HBase](https://github.com/mravi/kafka-connect-hbase)\n* [FileSystem](https://github.com/mmolimar/kafka-connect-fs)\n* [Streaming files from a local filesystem](https://github.com/streamthoughts/kafka-connect-file-pulse)\n* [kafka-connect-protobuf-converter](https://github.com/blueapron/kafka-connect-protobuf-converter)\n* [IBM MQ Source](https://github.com/ibm-messaging/kafka-connect-mq-source)\n* [IBM MQ Sink](https://github.com/ibm-messaging/kafka-connect-mq-sink)\n* [Azure IoT Hub](https://github.com/Azure/toketi-kafka-connect-iothub)\n* [Pulsar](https://github.com/streamnative/pulsar-io-kafka)\n* [HTTP](https://github.com/thomaskwscott/kafka-connect-http)\n\n### Producers\n* [Maxwell's daemon - a mysql-to-json kafka producer](https://github.com/zendesk/maxwell)\n\n### Consumers\n* [Spark](https://github.com/dibbhatt/kafka-spark-consumer)\n* [Storm](http://storm.apache.org/)\n* [Mapreduce](https://github.com/Conductor/kangaroo)\n* [Divolte](https://github.com/divolte/divolte-kafka-consumer)\n* [Azure Data Explorer/Kusto](https://github.com/Azure/kafka-sink-azure-kusto)\n* [Firehose](https://github.com/odpf/firehose) - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.\n* [Raccoon](https://github.com/odpf/raccoon) - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka.\n\n### Transformations\n* [Dagger](https://github.com/odpf/dagger) - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data.\n* [Single Message Transformations (SMT)](https://github.com/aiven/aiven-kafka-connect-transforms)\n\n### Testing\n* [Mocked Streams - Topology testing library for Kafka Streams (no broker required)](https://github.com/jpzk/mockedstreams)\n* [Unit-testing for embedded kafka services](https://github.com/Landoop/kafka-testing)\n* [Kafka JUnit](https://github.com/salesforce/kafka-junit)\n* [Kafka Unit](https://github.com/chbatey/kafka-unit)\n* [Kafka for JUnit](https://github.com/mguenther/kafka-junit)\n* [Karate and Kafka](https://github.com/karatelabs/karate-examples/blob/main/kafka/README.md)\n* [embedded-kafka - in-memory Kafka instance to run your tests against](https://github.com/embeddedkafka/embedded-kafka)\n* [Kafka Containers](https://www.testcontainers.org/modules/kafka/)\n* [Mockintosh - through asynchronous Kafka actors](https://mockintosh.io/)\n\n\u003c!--- ### Managed Services  --\u003e\n## Operations\n\n### CI/CD\n* [kafka-gitops](https://github.com/devshawn/kafka-gitops)\n\n### Operational Utilities\n* [Cruise control](https://github.com/linkedin/cruise-control)\n* [DoctorKafka](https://github.com/pinterest/doctorkafka/) and its successor [Orion](https://github.com/pinterest/orion)\n* [Brucke - Inter-cluster bridge of kafka topics](https://github.com/klarna/brucke)\n* [Kafka-Kit](https://github.com/DataDog/kafka-kit)\n* [Secor - service for persisting Kafka logs](https://github.com/pinterest/secor)\n* [kafkabalancer](https://github.com/CAFxX/kafkabalancer)\n* [Yelp Kafka-Utils](https://github.com/Yelp/kafka-utils)\n* [Cerner Common Kafka utilities](https://github.com/cerner/common-kafka)\n* [connectctl - manage kafka connect connectors easily](https://github.com/90poe/connectctl)\n\n### Monitoring\n* [Kafka Offset Exporter](https://github.com/echojc/kafka-offset-exporter)\n* [Fluent plugin](https://github.com/fluent/fluent-plugin-kafka)\n* [Burrow - Kafka Consumer Lag Checking](https://github.com/linkedin/Burrow)\n* [Remora](https://github.com/zalando-incubator/remora)\n* [Kafka Offset Monitoring](https://github.com/Morningstar/kafka-offset-monitor)\n* [kafka-eagle](https://github.com/smartloli/kafka-eagle)\n* [Kafka-prometheus](https://github.com/streamthoughts/kafka-monitoring-suite-demo-prometheus)\n* [Consumer lag monitoring](https://github.com/omarsmak/kafka-consumer-lag-monitoring)\n\n### Performance tools\n* [sangrenel](https://github.com/jamiealquiza/sangrenel)\n* [KLoadGen - load generator plugin for jmeter](https://github.com/corunet/kloadgen)\n\n### Security\n* [Kafka Security Manager](https://github.com/simplesteph/kafka-security-manager)\n* [Kafka security playbook](https://github.com/Dabz/kafka-security-playbook)\n\n### Audit\n* [chaperone - A Kafka audit system](https://github.com/uber/chaperone)\n\n### Mirroring\n* [MirrorMaker](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330)\n* [Message Handler For MirrorMaker to change topic names](https://github.com/opencore/mirrormaker_topic_rename)\n* [uReplicator](https://github.com/uber/uReplicator)\n* [MirrorTool for Kafka Connect](https://github.com/Comcast/MirrorTool-for-Kafka-Connect)\n* [Mirus](https://github.com/salesforce/mirus)\n\n### Backup\n* [Backup and Restore topics \u0026 offsets](https://github.com/itadventurer/kafka-backup)\n\n### Tools\n\n* [Karapace - Your Kafka essentials in one tool](https://github.com/aiven/karapace)\n* [REST-proxy](https://github.com/confluentinc/kafka-rest)\n* [Kafka-Pixy - gRPC/REST Proxy](https://github.com/mailgun/kafka-pixy)\n* [Kafka Proxy providing SASL/PLAIN authentication and SSL](https://github.com/grepplabs/kafka-proxy)\n* [Reactive/Akka API](https://github.com/akka/reactive-kafka)\n* [kafkat](https://github.com/airbnb/kafkat)\n* [kt](https://github.com/fgeller/kt)\n* [kafka-connect-tools](https://github.com/datamountaineer/kafka-connect-tools)\n* [Node Kafka connect](https://github.com/nodefluent/kafka-connect)\n* [Kafka-monitor](https://github.com/linkedin/kafka-monitor)\n* [Kafka Auto completion](https://github.com/Landoop/kafka-autocomplete)\n* [Kafka Cheet Sheet](https://github.com/Landoop/kafka-cheat-sheet)\n* [Kafka Write Protocol encoder/decoder](https://github.com/klarna/kafka_protocol)\n* [Winton Kafka Streams](https://github.com/wintoncode/winton-kafka-streams)\n* [Spring-kafka](https://github.com/spring-projects/spring-kafka)\n* [Maven quick start for Kafka Connect connectors](https://github.com/jcustenborder/kafka-connect-archtype)\n* [Trifecta](https://github.com/ldaniels528/trifecta)\n* [Kafka Specs](https://github.com/streamthoughts/kafka-specs)\n* [Strimzi Kafka CLI](https://github.com/systemcraftsman/strimzi-kafka-cli)\n\n### Metadata Management\n* [Go library for Kafka metadata in Zookeeper](https://github.com/wvanbergen/kazoo-go)\n* [Meteor](https://github.com/odpf/meteor) - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka.\n\n### Schema Management\n* [Stencil](https://github.com/odpf/stencil) - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably.\n* [Schema registry](https://github.com/confluentinc/schema-registry)\n* [Node Schema registry](https://github.com/nodefluent/schema-registry)\n* [Schema Registry Transfer SMT](https://github.com/cricket007/schema-registry-transfer-smt)\n* [Expedia Stream Registry](https://github.com/ExpediaGroup/stream-registry)\n\n### UI\n* [kafka-manager](https://github.com/yahoo/kafka-manager)\n* [Kafka HQ](https://github.com/tchiotludo/kafkahq)\n* [Kafdrop 3.x](https://github.com/obsidiandynamics/kafdrop) – web UI for Kafka. For versions older than 0.10.0 use [Kafdrop 2.x](https://github.com/HomeAdvisor/Kafdrop).\n* [Kafka Webview](https://github.com/SourceLabOrg/kafka-webview)\n* [Kafka Topics UI](https://github.com/Landoop/kafka-topics-ui)\n* [Kafka Connect UI](https://github.com/Landoop/kafka-connect-ui)\n* [Kafka Schema UI](https://github.com/Landoop/schema-registry-ui)\n* [Kafkawize - self service topic management portal](https://github.com/muralibasani/kafkawize)\n* [Kadmin - Kafka Producer/Consumer UI](https://github.com/BetterCloud/kadmin)\n* [Kowl](https://github.com/cloudhut/kowl)\n* [Kafka Streams Topology Sketch Diagram Visualization](https://github.com/zz85/kafka-streams-viz)\n* [Web based GUI](https://github.com/provectus/kafka-ui)\n\n## Deployment\n\n### Kubernetes\n* [Kafka k8s operator](https://github.com/krallistic/kafka-operator)\n* [Strimzi](https://github.com/strimzi/strimzi-kafka-operator) Operator for deploying and running Apache Kafka on Kubernetes\n* [konsumerator - Kafka Consumer Operator](https://github.com/lwolf/konsumerator)\n\n### Docker\n* [fast-data-dev](https://github.com/Landoop/fast-data-dev)\n\n### Integrations\n\n* [Ecosystem](https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem)\n* [hermes](https://github.com/allegro/hermes)\n* [IBMStreams](https://github.com/IBMStreams/streamsx.kafka)\n* [Brooklin](https://github.com/linkedin/brooklin)\n\n## Alternative Implementations\n\n* [Go](https://github.com/travisjeffery/jocko)\n* [C/C++](https://github.com/edenhill/librdkafka)\n\n## Projects built on top of Kafka\n* [In-Memory Cache Backed by Kafka](https://github.com/rayokota/kcache)\n* [ransactional Metadata Store Backed by Kafka](https://github.com/rayokota/keta)\n\n## Unsorted\n\n* https://github.com/Landoop/kafka-connect-kcql-smt\n* https://github.com/nodefluent/purpur\n* [Azkarra Streams - lightweight java framework which makes easy to create production-ready Kafka Streams applications](https://github.com/streamthoughts/azkarra-streams)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdharmeshkakadia%2Fawesome-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdharmeshkakadia%2Fawesome-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdharmeshkakadia%2Fawesome-kafka/lists"}