awesome-kafka
Everything about Apache Kafka
https://github.com/dharmeshkakadia/awesome-kafka
Last synced: 5 days ago
JSON representation
Alternative Implementations
Deployment
Docker
Integrations
Kubernetes
- Kafka k8s operator
- Strimzi - Operator for deploying and running Apache Kafka on Kubernetes
- konsumerator - Kafka Consumer Operator
Development
Client libraries
- Go
- Go
- Go Kafka streaming
- Node
- Node
- Node Kafka streams
- Node librdkafka
- Php
- Php
- Perl
- Python (asyncio)
- Python (kafka-python)
- Python (pykafka)
- Ruby
- Karafka - Ruby and Rails framework
- Dart
- JRuby
- Elixir
- LinkedIn wrapper library
- Go
- Rust
- kasper - a lightweight library for processing Kafka topics
- Node
- Phobos - Ruby App framework
- Erlang
- others
Connectors
- JDBC
- Elasticsearch
- HDFS
- BigQuery
- SQL
- GCloud pubsub
- BigTable
- HBase
- FileSystem
- Streaming files from a local filesystem
- kafka-connect-protobuf-converter
- IBM MQ Source
- IBM MQ Sink
- Azure IoT Hub
- Pulsar
- HTTP
- Beast - Kafka to BigQuery Sink
- Google Cloud Storage (GCS)
Consumers
- Spark
- Mapreduce
- Divolte
- Azure Data Explorer/Kusto
- Storm
- Raccoon - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka.
Kafka Streams Libraries
KSQL
Producers
Testing
- Karate and Kafka
- Mockintosh - through asynchronous Kafka actors
- Mocked Streams - Topology testing library for Kafka Streams (no broker required)
- Kafka JUnit
- Kafka Unit
- Kafka for JUnit
- embedded-kafka - in-memory Kafka instance to run your tests against
- Unit-testing for embedded kafka services
- Kafka Containers
Transformations
- Dagger - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data.
Operations
Backup
CI/CD
- kafka-gitops
- cdclint - lint a Debezium connector config against Postgres migrations and the sink schema in CI
Metadata Management
- Go library for Kafka metadata in Zookeeper
- Meteor - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka.
Mirroring
- Message Handler For MirrorMaker to change topic names
- uReplicator
- MirrorTool for Kafka Connect
- Mirus
Monitoring
- Kafka Offset Exporter
- Fluent plugin
- Burrow - Kafka Consumer Lag Checking
- Remora
- Kafka Offset Monitoring
- Consumer lag monitoring
- Kafka-prometheus
- kafka-eagle
- SolidPing - self-hosted uptime monitor with a Kafka broker and topic check
Operational Utilities
- Cruise control
- Brucke - Inter-cluster bridge of kafka topics
- Kafka-Kit
- Secor - service for persisting Kafka logs
- kafkabalancer
- Yelp Kafka-Utils
- Cerner Common Kafka utilities
- connectctl - manage kafka connect connectors easily
- DoctorKafka - and its successor Orion
Performance tools
Schema Management
- Schema registry
- Node Schema registry
- Expedia Stream Registry
- Stencil - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably.
- Schema Registry Transfer SMT
Security
Tools
- REST-proxy
- Kafka-Pixy - gRPC/REST Proxy
- Kafka Proxy providing SASL/PLAIN authentication and SSL
- kafkat
- kt
- Node Kafka connect
- Kafka-monitor
- Winton Kafka Streams
- Spring-kafka
- Maven quick start for Kafka Connect connectors
- Trifecta
- Kafka Write Protocol encoder/decoder
- Kafka Cheet Sheet
- Kafka Specs
- Reactive/Akka API
- Strimzi Kafka CLI
- Karapace - Your Kafka essentials in one tool
- kafka-connect-tools
- Kafka Auto completion
UI
- Kafdrop 3.x - web UI for Kafka. For versions older than 0.10.0 use Kafdrop 2.x.
- Kafka Webview
- Kafkawize - self service topic management portal
- Kadmin - Kafka Producer/Consumer UI
- Kafka Streams Topology Sketch Diagram Visualization
- Web based GUI
- Kowl
- Kafka Topics UI
- Kafka Schema UI
- Kafka HQ
- LibreDB Studio - web-based multi-database IDE with a read-only Kafka browser for topics, consumer group lag, brokers and messages by offset or timestamp
- Kafka Connect UI