Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with kafka

A curated list of projects in awesome lists tagged with kafka .

https://github.com/datatalksclub/data-engineering-zoomcamp

Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.

data-engineering dbt docker kafka kestra spark

Last synced: 13 May 2025

https://github.com/apache/kafka

Mirror of Apache Kafka

kafka scala

Last synced: 12 May 2025

https://github.com/DataTalksClub/data-engineering-zoomcamp

Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.

data-engineering dbt docker kafka kestra spark

Last synced: 14 Mar 2025

https://github.com/yudaocode/springboot-labs

一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024

activemq dubbo kafka rabbitmq rocketmq spring-boot spring-cloud spring-cloud-alibaba

Last synced: 13 May 2025

https://github.com/thingsboard/thingsboard

Open-source IoT Platform - Device management, data collection, processing and visualization.

cloud coap dashboard iot iot-analytics iot-platform iot-solutions java kafka lwm2m microservices middleware mqtt netty platform snmp thingsboard visualization websockets widgets

Last synced: 12 May 2025

https://github.com/yudaocode/SpringBoot-Labs

一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024

activemq dubbo kafka rabbitmq rocketmq spring-boot spring-cloud spring-cloud-alibaba

Last synced: 24 Mar 2025

https://github.com/influxdata/telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

gnmi golang influxdb json kafka logs metrics modbus monitoring mqtt opcua telegraf time-series windows-eventlog windows-management-instrumentation xpath

Last synced: 12 May 2025

https://github.com/zhisheng17/flink-learning

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

clickhouse elasticsearch flink hbase influxdb kafka loki mysql opentsdb rabbitmq redis rocketmq spark stream-processing streaming

Last synced: 14 May 2025

https://github.com/aalansehaiyang/technology-talk

【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让你成为更牛的自己！

dubbo es6 git hbase java kafka mycat spark spring springboot

Last synced: 13 May 2025

https://github.com/sogou/workflow

C++ Parallel Computing and Asynchronous Networking Framework

consul dag http kafka mysql redis sogou tasking

Last synced: 14 May 2025

https://github.com/ibm/sarama

Sarama is a Go library for Apache Kafka.

go kafka kafka-client

Last synced: 12 May 2025

https://github.com/yahoo/cmak

CMAK is a tool for managing Apache Kafka clusters

big-data cluster-management kafka scala

Last synced: 13 May 2025

https://github.com/yahoo/CMAK

CMAK is a tool for managing Apache Kafka clusters

big-data cluster-management kafka scala

Last synced: 03 Apr 2025

https://github.com/debezium/debezium

Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

apache-kafka cdc change-data-capture data-pipeline database debezium event-streaming kafka kafka-connect kafka-producer

Last synced: 12 May 2025

https://github.com/kubeshark/kubeshark

The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes

amqp cloud-native devops devops-tools docker forensics go golang grpc incident-response kafka kubernetes microservice microservices microservices-application observability redis rest sniffer wireshark

Last synced: 13 May 2025

https://github.com/IBM/sarama

Sarama is a Go library for Apache Kafka.

go kafka kafka-client

Last synced: 14 Mar 2025

https://github.com/redpanda-data/redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

containers cpp event-driven go kafka kubernetes microservices realtime redpanda seastar storage-engine stream-processing streaming

Last synced: 13 May 2025

https://github.com/wangzhiwubigdata/god-of-bigdata

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

azkaban bigdata flink flume hadoop hbase hdfs hive kafka spark zookeeper

Last synced: 13 May 2025

https://github.com/wangzhiwubigdata/God-Of-BigData

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

azkaban bigdata flink flume hadoop hbase hdfs hive kafka spark zookeeper

Last synced: 27 Mar 2025

https://github.com/didi/knowstreaming

一站式云原生实时流数据平台，通过0侵入、插件化构建企业级Kafka服务，极大降低操作、存储和管理实时流数据门槛

cluster-manager clustermanager kafka kafkamanager kafkamonitor knowstreaming logikm multi-cluster multi-cluster-management

Last synced: 23 Apr 2025

https://github.com/didi/KnowStreaming

一站式云原生实时流数据平台，通过0侵入、插件化构建企业级Kafka服务，极大降低操作、存储和管理实时流数据门槛

cluster-manager clustermanager kafka kafkamanager kafkamonitor knowstreaming logikm multi-cluster multi-cluster-management

Last synced: 24 Mar 2025

https://github.com/RisingWaveLabs/risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.

analytics big-data cloud-native data-engineering database distributed-database etl flink kafka ksqldb materialized-view postgres postgresql real-time real-time-analytics rust serverless spark-streaming sql stream-processing

Last synced: 29 Mar 2025

https://github.com/dotnetcore/cap

Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

distributed-transactions eventbus kafka microservices outbox-pattern rabbitmq service-bus

Last synced: 13 May 2025

https://github.com/dotnetcore/CAP

Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

distributed-transactions eventbus kafka microservices outbox-pattern rabbitmq service-bus

Last synced: 16 Mar 2025

https://github.com/harisekhon/devops-bash-tools

1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..

api aws bash ci cloudera devops docker gcp git github hacktoberfest hadoop jenkins kafka kubernetes linux mysql perl postgresql terraform

Last synced: 23 Apr 2025

https://github.com/HariSekhon/DevOps-Bash-tools

1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..

api aws bash ci cloudera devops docker gcp git github hacktoberfest hadoop jenkins kafka kubernetes linux mysql perl postgresql terraform

Last synced: 02 Apr 2025

https://github.com/coderleixiaoshuai/java-eight-part

『Java八股文』Java面试套路，Java进阶学习，打破内卷拿大厂Offer，升职加薪！

java juc kafka mysql redis spring

Last synced: 14 May 2025

https://github.com/CoderLeixiaoshuai/java-eight-part

『Java八股文』Java面试套路，Java进阶学习，打破内卷拿大厂Offer，升职加薪！

java juc kafka mysql redis spring

Last synced: 10 Apr 2025

https://github.com/materializeinc/materialize

Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.

data-store database distributed-systems kafka materialized-view operational-data-store postgresql postgresql-dialect rust sql stream-processing streaming streaming-data

Last synced: 13 May 2025

https://github.com/opengoofy/hippo4j

📌 异步线程池框架，支持线程池动态变更&监控&报警，无需修改代码轻松引入。Asynchronous thread pool framework, support Thread Pool Dynamic Change & monitoring & Alarm, no need to modify the code easily introduced.

blocking-queue dynamic dynamic-threadpool java kafka monitor resizable rocketmq springboot thread-pool

Last synced: 14 May 2025

https://github.com/dpkp/kafka-python

Python client for Apache Kafka

kafka python

Last synced: 12 May 2025

https://github.com/rfyiamcool/share_ppt

🚗 个人做过的技术分享... ( topic: 高性能架构, 中间件原理实践，云原生，Golang 等)

etcd golang kafka keynote kubernetes mysql ppt python

Last synced: 14 May 2025

https://github.com/automq/automq

AutoMQ is a stateless Kafka on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

ai apache-kafka aws azure cloud cloud-first cloud-native ebs gcp kafka llm messaging minio s3 serverless spot streaming

Last synced: 12 May 2025

https://github.com/travisjeffery/jocko

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

distributed-systems go kafka messaging queue streaming

Last synced: 14 May 2025

https://github.com/2227324689/gpmall

【咕泡学院实战项目】-基于SpringBoot+Dubbo构建的电商平台-微服务架构、商城、电商、微服务、高并发、kafka、Elasticsearch

canal dubbo elasticsearch elk java kafka nacos sentinel springboot springcloud-alibaba zookeeper

Last synced: 14 May 2025

https://github.com/mikaelemmmm/go-zero-looklook

🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.

asynq dtm elasticsearch filebeat gitlab go-queue go-stash go-zero goctl golang gozero grafana harbor jaeger jenkins k8s kafka microservices modd prometheus

Last synced: 13 May 2025

https://github.com/AutoMQ/automq

AutoMQ is a cloud-native alternative to Kafka by decoupling durability to cloud storage services like S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

apache-kafka aws azure cloud cloud-economics cloud-first cloud-native ebs gcp kafka messaging minio s3 serverless spot storage streaming

Last synced: 31 Mar 2025

https://github.com/Mikaelemmmm/go-zero-looklook

🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.

asynq dtm elasticsearch filebeat gitlab go-queue go-stash go-zero goctl golang gozero grafana harbor jaeger jenkins k8s kafka microservices modd prometheus

Last synced: 10 Apr 2025

https://github.com/asyncapi/spec

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

amqp asyncapi hacktoberfest kafka mqtt reactive specification stomp websockets

Last synced: 14 May 2025

https://github.com/build-trust/ockam

Orchestrate end-to-end encryption, cryptographic identities, mutual authentication, and authorization policies between distributed applications – at massive scale.

authentication authorization credentials distributed-systems e2ee encrypted-connections encrypted-messages encryption end-to-end-encryption identity kafka key-management messaging rust security snowflake trust zero-trust

Last synced: 12 May 2025

https://github.com/redpanda-data/console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.

apache-kafka dataops go kafka kafka-gui kafka-ui react typescript web-ui

Last synced: 14 May 2025

https://github.com/ag2ai/faststream

FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing

Last synced: 14 May 2025

https://github.com/tulios/kafkajs

A modern Apache Kafka client for node.js

kafka kafka-client nodejs npm-package

Last synced: 11 May 2025

https://github.com/nytimes/gizmo

A Microservice Toolkit from The New York Times

gizmo go google-pubsub googlecloud http-server kafka pubsub sns sqs

Last synced: 14 May 2025

https://github.com/NYTimes/gizmo

A Microservice Toolkit from The New York Times

gizmo go google-pubsub googlecloud http-server kafka pubsub sns sqs

Last synced: 06 May 2025

https://github.com/alibaba/alink

Alink is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform.

apriori classification clustering data-mining feature-engineering flink flink-machine-learning flink-ml fm graph-algorithms graph-embedding kafka machine-learning recommender recommender-system regression statistics word2vec xgboost

Last synced: 14 May 2025

https://github.com/alibaba/Alink

Alink is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform.

apriori classification clustering data-mining feature-engineering flink flink-machine-learning flink-ml fm graph-algorithms graph-embedding kafka machine-learning recommender recommender-system regression statistics word2vec xgboost

Last synced: 14 Mar 2025

https://github.com/tchiotludo/akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...

gui java kafka kafka-tools kafka-ui kafka-utils kafkahq

Last synced: 13 May 2025

https://github.com/dotnetcore/surging

Surging is a micro-service engine that provides a lightweight, high-performance, modular RPC request pipeline. support Event-based Asynchronous Pattern and reactive programming ,The service engine supports http, TCP, WS,Grpc, Thrift,Mqtt, UDP, and DNS protocols. It uses ZooKeeper and Consul as a registry, and integrates it. Hash, random, polling, Fair Polling as a load balancing algorithm, built-in service governance to ensure reliable RPC communication, the engine contains Diagnostic, link tracking for protocol and middleware calls, and integration SkyWalking Distributed APM

apigateway apollo consul dns-server grpc json-serialization kafka kestrel log4net memorycache messagepack microservices nlog protobuffer rabbitmq redis skywalking surging swagger zookeeper

Last synced: 06 Feb 2025

https://github.com/fanliang11/surging

Surging is a micro-service engine that provides a lightweight, high-performance, modular RPC request pipeline. support Event-based Asynchronous Pattern and reactive programming ,The service engine supports http, TCP, WS,Grpc, Thrift,Mqtt, UDP, and DNS protocols. It uses ZooKeeper and Consul as a registry, and integrates it. Hash, random, polling, Fair Polling as a load balancing algorithm, built-in service governance to ensure reliable RPC communication, the engine contains Diagnostic, link tracking for protocol and middleware calls, and integration SkyWalking Distributed APM

apigateway apollo consul dns-server grpc json-serialization kafka kestrel log4net memorycache messagepack microservices nlog protobuffer rabbitmq redis skywalking surging swagger zookeeper

Last synced: 14 May 2025

https://github.com/smartloli/efak

A easy and high-performance monitoring system, for comprehensive monitoring and management of kafka cluster.

consumer-group kafka kafka-eagle kafka-sql ksql offsets zookeeper

Last synced: 15 May 2025

https://github.com/smartloli/EFAK

A easy and high-performance monitoring system, for comprehensive monitoring and management of kafka cluster.

consumer-group kafka kafka-eagle kafka-sql ksql offsets zookeeper

Last synced: 27 Mar 2025

https://github.com/sanshengshui/netty-learning-example

:egg: Netty实践学习案例，见微知著！带着你的心，跟着教程。我相信你行欧。

fastjson http im iot kafka mqtt netty protobuf rpc zero-copy

Last synced: 15 May 2025

https://github.com/memgraph/memgraph

Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

cypher graph graph-algorithms graph-analysis graph-database kafka kafka-streams nosql opencypher stream-processing streaming-data

Last synced: 14 May 2025

https://github.com/moran1607/bigdataguide

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

bigdata flink flume hadoop hbase hive javase kafka scala spark zookeeper

Last synced: 14 May 2025

https://github.com/MoRan1607/BigDataGuide

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

bigdata flink flume hadoop hbase hive javase kafka scala spark zookeeper

Last synced: 04 Apr 2025

https://github.com/linkedin/cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.

cluster-management kafka self-healing

Last synced: 11 May 2025

https://github.com/qunarcorp/qmq

QMQ是去哪儿网内部广泛使用的消息中间件，自2012年诞生以来在去哪儿网所有业务场景中广泛的应用，包括跟交易息息相关的订单场景； 也包括报价搜索等高吞吐量场景。

consumer distributed-transactions event event-bus event-driven kafka message message-bus message-driven message-queue mq producer rabbitmq xa

Last synced: 14 May 2025

https://github.com/airtai/faststream

FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing

Last synced: 25 Mar 2025

https://github.com/apecloud/kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.

database dbaas dbpaas kafka kubernetes mongodb mysql operator postgresql redis

Last synced: 12 May 2025

https://github.com/cosen1024/java-interview

「Java面试小抄」一份通向理想互联网公司的面试汇总，包括 Java基础、Java并发、JVM、MySQL、Redis、Spring、MyBatis、Kafka、计算机操作系统、计算机网络、系统设计、分布式、Java 项目实战等

dubbo interview java jvm kafka mysql redis spring

Last synced: 15 May 2025

https://github.com/peerdb-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 14 May 2025

https://github.com/lovoo/goka

Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.

goka golang kafka microservices processing stream

Last synced: 13 May 2025

https://github.com/PeerDB-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 28 Mar 2025

https://github.com/syslog-ng/syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

c elastic kafka log-management logging python syslog syslog-ng

Last synced: 14 May 2025

https://github.com/balabit/syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

c elastic kafka log-management logging python syslog syslog-ng

Last synced: 11 Mar 2025

https://github.com/php-enqueue/enqueue-dev

Message Queue, Job Queue, Broadcasting, WebSockets packages for PHP, Symfony, Laravel, Magento. DEVELOPMENT REPOSITORY - provided by Forma-Pro

amazon-sqs amqp beanstalkd doctrine-dbal gearman google-pubsub hacktoberfest job-queue kafka message-bus message-passing message-queue rabbitmq rabbitmq-client redis sns stomp symfony wamp wamp-client

Last synced: 14 May 2025

https://github.com/blizzard/node-rdkafka

Node.js bindings for librdkafka

kafka librdkafka nodejs

Last synced: 11 May 2025

https://github.com/Blizzard/node-rdkafka

Node.js bindings for librdkafka

kafka librdkafka nodejs

Last synced: 25 Mar 2025