Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with kafka-connect

A curated list of projects in awesome lists tagged with kafka-connect .

https://github.com/debezium/debezium

Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

apache-kafka cdc change-data-capture database debezium event-streaming kafka kafka-connect kafka-producer

Last synced: 16 Dec 2024

https://github.com/confluentinc/demo-scene

👾Scripts and samples to support Confluent Demos and Talks up until Oct '24. ⚠️ No longer maintained 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/

debezium demo elasticsearch kafka kafka-connect kafka-streams ksql ksqldb mysql syslog

Last synced: 19 Dec 2024

https://github.com/xaecbd/KCenter

KCenter(KafkaCenter) is a unified platform for kafka cluster management and maintenance, producer / consumer monitoring, and use of ecological components(ksql/kafka connect).it's kafkacenter(kafka center).

cmak connect-ui kafka kafka-center kafka-connect kafka-connect-ui kafka-eagle kafka-manager kafka-monitor kafkacenter kafkamanager kafkaoffsetmonitor kc kmanager ksql

Last synced: 06 Nov 2024

https://github.com/xaecbd/kcenter

KCenter(KafkaCenter) is a unified platform for kafka cluster management and maintenance, producer / consumer monitoring, and use of ecological components(ksql/kafka connect).it's kafkacenter(kafka center).

cmak connect-ui kafka kafka-center kafka-connect kafka-connect-ui kafka-eagle kafka-manager kafka-monitor kafkacenter kafkamanager kafkaoffsetmonitor kc kmanager ksql

Last synced: 15 Dec 2024

https://github.com/lensesio/stream-reactor

A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.

awss3 azure-datalake cassandra connector cosmosdb elasticsearch eventhubs gcp http influxdb jms kafka kafka-connect mongodb mqtt pubsub s3

Last synced: 19 Dec 2024

https://github.com/vdesabou/kafka-docker-playground

🐳✨ Fully automated Apache Kafka® and Confluent Docker based examples // 👷‍♂️ Easily build examples or reproduction models

confluent confluent-cloud confluent-kafka confluent-platform kafka kafka-connect kafka-connectors

Last synced: 20 Dec 2024

https://github.com/ddd-by-examples/all-things-cqrs

Comprehensive guide to a couple of possible ways of synchronizing two states with Spring tools. Synchronization is shown by separating command and queries in a simple CQRS application.

cqrs ddd debezium domain-driven-design events kafka kafka-connect mongodb spring spring-boot

Last synced: 15 Dec 2024

https://github.com/langstream/langstream

LangStream. Event-Driven Developer Platform for Building and Running LLM AI Apps. Powered by Kubernetes and Kafka.

genai gpt-4 huggingface kafka kafka-connect kubernetes langchain llamaindex llm openai python vectordb

Last synced: 20 Dec 2024

https://github.com/LangStream/langstream

LangStream. Event-Driven Developer Platform for Building and Running LLM AI Apps. Powered by Kubernetes and Kafka.

genai gpt-4 huggingface kafka kafka-connect kubernetes langchain llamaindex llm openai python vectordb

Last synced: 06 Nov 2024

https://github.com/streamthoughts/kafka-connect-file-pulse

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

amazon-s3 avro azure-storage csv etl file-streaming google-cloud grok-filters kafka kafka-connect kafka-connector kafka-producer xml

Last synced: 13 Nov 2024

https://github.com/Altinity/clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

avro cdc clickhouse debezium kafka kafka-connect mongo mysql postgresql replication sqlserver

Last synced: 12 Nov 2024

https://github.com/altinity/clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

avro cdc clickhouse debezium kafka kafka-connect mongo mysql postgresql replication sqlserver

Last synced: 15 Dec 2024

https://github.com/salesforce/mirus

Mirus is a cross data-center data replication tool for Apache Kafka

kafka kafka-connect mirroring

Last synced: 17 Dec 2024

https://github.com/apecloud/ape-dts

Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios.

binlog cdc clickhouse data-migration data-transfer etl kafka-connect mongo mysql postgres redis starrocks

Last synced: 16 Dec 2024

https://github.com/jcustenborder/kafka-connect-spooldir

Kafka Connect connector for reading CSV files into Kafka.

csv json kafka-connect

Last synced: 21 Dec 2024

https://github.com/itadventurer/kafka-backup

Backup and Restore for Apache Kafka

backup kafka kafka-connect

Last synced: 21 Dec 2024

https://github.com/jcustenborder/kafka-connect-transform-common

Common Transforms for Kafka Connect.

kafka-connect transformations

Last synced: 15 Dec 2024

https://github.com/hpgrahsl/kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

avro azure-cosmosdb bson cdc change-data-capture confluent-hub connector cosmosdb debezium json kafka kafka-connect mongodb sink sink-connector

Last synced: 18 Nov 2024

https://github.com/aiven-open/klaw

Klaw, the latest OS tool by Aiven, helps enterprises cope with Apache Kafka(r) topics, schema registry and connectors governance by introducing roles/authorizations to users of various teams of an org.

apache-kafka governance kafka kafka-connect kafka-governance

Last synced: 21 Dec 2024

https://github.com/mtpatter/postgres-kafka-demo

Fully reproducible, Dockerized, step-by-step, demo on how to stream tables from Postgres to Kafka/KSQL back to Postgres. Detailed blog post published on Medium.

debezium kafka kafka-connect ksql ksql-server postgresql streaming

Last synced: 13 Nov 2024

https://github.com/snowflakedb/snowflake-kafka-connector

Snowflake Kafka Connector (Sink Connector)

java kafka kafka-connect snowflake

Last synced: 13 Nov 2024

https://github.com/nodefluent/kafka-connect

equivalent to kafka-connect :wrench: for nodejs :sparkles::turtle::rocket::sparkles:

connect datastore etl framework kafka kafka-connect nodejs

Last synced: 18 Dec 2024

https://github.com/jcustenborder/kafka-connect-twitter

Kafka Connect connector to stream data in real time from Twitter.

kafka-connect twitter

Last synced: 20 Dec 2024

https://github.com/lensesio/kafka-connect-tools

Kafka Connect Tooling

cli kafka kafka-connect

Last synced: 17 Nov 2024

https://github.com/mongey/terraform-provider-kafka-connect

Terraform provider for managing Apache Kafka Connect

kafka kafka-connect terraform terraform-provider

Last synced: 15 Dec 2024

https://github.com/ivangfr/springboot-kafka-connect-jdbc-streams

Project goal: Explore Kafka, Kafka Connect, and Kafka Streams. Components: store-api: Inserts/updates MySQL records. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors: Listen to Kafka, insert/update Elasticsearch docs. store-streams: Listens to Kafka, processes with Kafka Streams, pushes new messages to Kafka.

avro docker elasticsearch java jib json kafka kafka-connect kafka-streams mapstruct mysql schema-registry spring-boot spring-cloud-stream spring-data-jpa spring-web-mvc springdoc-openapi

Last synced: 19 Dec 2024

https://github.com/kakao/kafka-connect-web

Web tool for operating kafka connect https://hub.docker.com/r/officialkakao/kafka-connect-web

kafka kafka-connect streaming

Last synced: 19 Nov 2024

https://github.com/aiven-open/http-connector-for-apache-kafka

Apache Kafka Connect sink connector for HTTP

http kafka kafka-connect

Last synced: 15 Dec 2024

https://github.com/ibm-messaging/kafka-connect-mq-source

This repository contains a Kafka Connect source connector for copying data from IBM MQ into Apache Kafka.

ibm-mq kafka-connect

Last synced: 21 Dec 2024

https://github.com/aiven-open/jdbc-connector-for-apache-kafka

Aiven's JDBC Sink and Source Connectors for Apache Kafka®

jdbc kafka kafka-connect

Last synced: 15 Dec 2024

https://github.com/hpgrahsl/kryptonite-for-kafka

Kryptonite for Kafka is a client-side 🔒 field level 🔓 cryptography library for Apache Kafka® offering a Kafka Connect SMT, ksqlDB UDFs, and a standalone HTTP API service. It's an ! UNOFFICIAL ! community project

apache-kafka cryptography encryption-decryption end-to-end-encryption field-level-encryption kafka kafka-connect ksqldb smt transformations udfs user-defined-functions

Last synced: 17 Nov 2024

https://github.com/ivangfr/springboot-kafka-connect-debezium-ksqldb

Experiment with Kafka, Debezium, and ksqlDB. research-service: Performs MySQL record manipulation. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors and kafka-research-consumer: Listen to Kafka, insert/update Elasticsearch. ksqlDB-Server: Listens to Kafka, performs joins, and pushes new messages to new Kafka topics.

avro confluent debezium elasticsearch java json kafka kafka-connect ksqldb mysql schema-registry spring-boot spring-data-jpa spring-kafka springdoc-openapi web

Last synced: 30 Oct 2024

https://github.com/Aiven-Open/transforms-for-apache-kafka-connect

Aiven's collection of Single Message Transformations (SMTs) for Apache Kafka Connect

kafka kafka-connect kafka-connect-transformations kafka-connect-transforms

Last synced: 13 Nov 2024

https://github.com/aiven-open/transforms-for-apache-kafka-connect

Aiven's collection of Single Message Transformations (SMTs) for Apache Kafka Connect

kafka kafka-connect kafka-connect-transformations kafka-connect-transforms

Last synced: 11 Nov 2024

https://github.com/aiven-open/s3-connector-for-apache-kafka

Aiven's S3 Sink Connector for Apache Kafka®

aws kafka kafka-connect s3

Last synced: 11 Nov 2024

https://github.com/niqdev/kafka-scala-examples

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

avro interactive-queries kafka kafka-connect kafka-scala kafka-streams ksql schema-registry

Last synced: 18 Dec 2024

https://github.com/garystafford/kafka-connect-msk-demo

For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR

aws kafka kafka-connect kubernetes spark spark-streaming

Last synced: 06 Dec 2024

https://github.com/jcustenborder/kafka-connect-redis

Kafka Connect connector for Redis

kafka-connect redis

Last synced: 20 Dec 2024

https://github.com/farmdawgnation/registryless-avro-converter

An avro converter for Kafka Connect without a Schema Registry

avro avro-schema kafka kafka-connect

Last synced: 03 Dec 2024

https://github.com/mguenther/kafka-junit

Enables you to start and stop a fully-fledged embedded Kafka cluster from within JUnit and provides a rich set of convenient accessors and fault injectors through a lean API. Supports working against external clusters as well.

junit kafka kafka-connect

Last synced: 17 Nov 2024

https://github.com/awslabs/eventbridge-kafka-connector

Kafka sink connector for Amazon EventBridge to send events (records) from Kafka topic(s) to the specified EventBridge event bus

event-driven-architecture eventbridge kafka kafka-connect

Last synced: 02 Nov 2024

https://github.com/onecricketeer/schema-registry-transfer-smt

Kafka Connect Transform to copy Avro schemas between Schema Registries

avro confluent kafka kafka-connect kafka-connect-smt schema-registry

Last synced: 20 Nov 2024

https://github.com/OneCricketeer/schema-registry-transfer-smt

Kafka Connect Transform to copy Avro schemas between Schema Registries

avro confluent kafka kafka-connect kafka-connect-smt schema-registry

Last synced: 17 Nov 2024

https://github.com/jaredpetersen/kafka-connect-redis

📕 Kafka Connect source and sink connector for Redis

kafka kafka-connect redis

Last synced: 18 Nov 2024

https://github.com/build-on-aws/building-apache-kafka-connectors

Sample code that shows the important aspects of developing custom connectors for Kafka Connect. It provides the resources for building, deploying, and running the code on-premises using Docker, as well as running the code in the cloud.

amazon-msk amazon-msk-connect apache-kafka data-streaming java kafka-connect kafka-connector terraform

Last synced: 07 Nov 2024

https://github.com/jcustenborder/kafka-connect-solr

Kafka Connect connector for writing to Solr.

kafka-connect solr

Last synced: 27 Oct 2024

https://github.com/tmcgrath/kafka-connect-examples

Kafka Connect Examples

kafka kafka-connect

Last synced: 05 Nov 2024

https://github.com/confluentinc/cp-ansible

Ansible playbooks for the Confluent Platform

ansible confluent kafka kafka-connect ksql playbook rest-proxy schema-registry

Last synced: 16 Nov 2024

https://github.com/nodefluent/sequelize-kafka-connect

:gem: nodejs kafka connect connector for MySQL, Postgres, SQLite and MSSQL

etl kafka kafka-connect mssql mysql nodejs postgres sequelize sqlite

Last synced: 11 Nov 2024

https://github.com/ibm-messaging/kafka-connect-mq-sink

This repository contains a Kafka Connect sink connector for copying data from Apache Kafka into IBM MQ.

ibm-mq kafka-connect

Last synced: 18 Dec 2024

https://github.com/SourceLabOrg/kafka-connect-client

A kafka-connect REST api client for java

java kafka kafka-connect rest-client

Last synced: 13 Nov 2024

https://github.com/jcustenborder/connect-utils

Utility project for working with Kafka Connect.

conversion kafka kafka-connect parsing schema

Last synced: 15 Dec 2024

https://github.com/ksindi/kafka-compose

:musical_score: Docker compose files for various kafka stacks

avro docker-compose kafka kafka-connect pyspark python spark twitter

Last synced: 12 Nov 2024

https://github.com/Anant/Cassandra.Realtime

Different ways to process data into Cassandra in realtime with technologies such as Kafka, Spark, Akka, Flink

akka cassandra flink flink-stream-processing flink-streaming kafka kafka-connect spark spark-streaming

Last synced: 08 Nov 2024

https://github.com/anant/cassandra.realtime

Different ways to process data into Cassandra in realtime with technologies such as Kafka, Spark, Akka, Flink

akka cassandra flink flink-stream-processing flink-streaming kafka kafka-connect spark spark-streaming

Last synced: 18 Nov 2024

https://github.com/yaravind/kafka-connect-jenkins

Kafka Connect Connector for Jenkins Open Source Continuous Integration Tool

jenkins kafka kafka-connect kafka-connector

Last synced: 13 Nov 2024

https://github.com/fhussonnois/kafkacli

CLI and Go Clients to manage Kafka components (Kafka Connect & SchemaRegistry)

cli-utilities kafka kafka-cli kafka-connect schema-registry

Last synced: 28 Oct 2024

https://github.com/jaredpetersen/kafka-connect-arangodb

🥑 Kafka connect sink connector for ArangoDB

arango arangodb kafka kafka-connect

Last synced: 18 Nov 2024

https://github.com/masesgroup/knet

KNet is a comprehensive .NET suite for Apache Kafka providing all features: Producer, Consumer, Admin, Streams, Connect, backends (ZooKeeper and Kafka).

apache-kafka dotnet java kafka kafka-client kafka-cluster kafka-connect kafka-connector kafka-consumer kafka-manager kafka-producer kafka-streams scala

Last synced: 24 Nov 2024

https://github.com/xushiyan/kafka-connect-datagen

A Kafka Connect source connector that generates data for tests

data-generator etl etl-pipeline integration-test java kafka kafka-connect performance-test

Last synced: 28 Oct 2024

https://github.com/helpermethod/connor

A commandline tool for resetting Kafka Connect source connector offsets.

graalvm kafka kafka-connect native-image

Last synced: 14 Oct 2024

https://github.com/jcustenborder/kafka-connect-splunk

Kafka Connect connector for receiving data and writing data to Splunk.

kafka-connect splunk splunk-http

Last synced: 28 Oct 2024

https://github.com/jcustenborder/kafka-connect-transform-xml

Transformation for converting XML data to Structured data.

kafka-connect transformation xml

Last synced: 28 Oct 2024

https://github.com/agoda-com/kafka-jdbc-connector

Simple way to copy data from relational databases into kafka.

jdbc-connector kafka-connect

Last synced: 06 Nov 2024

https://github.com/amadeusitgroup/kubernetes-kafka-connect-operator

A kubernetes operator to deploy and auto-scale KafkaConnect Application.

autoscaling golang kafka-connect kubernetes kubernetes-operator

Last synced: 10 Nov 2024

https://github.com/1ambda/docker-kafka-connect

Docker Image for kafka-connect

docker kafka-connect

Last synced: 18 Nov 2024

https://github.com/nodefluent/bigquery-kafka-connect

:cloud: nodejs kafka connect connector for Google BigQuery

big-data bigquery connect etl google-cloud kafka kafka-connect nodejs

Last synced: 11 Nov 2024

https://github.com/ably-labs/realtime-ticket-booking-solution

A simple demo showing how to use Ably and fastAPI to route messages into Kafka for stream processing

ably confluent-platform fastapi kafka kafka-connect ksqldb python

Last synced: 13 Nov 2024

https://github.com/hpgrahsl/kafka-streams-emojitracker

Sample application for my talk 'Building Stateful Streaming Applications Without a Database'

emojis kafka kafka-connect kafka-streams reactive-programming reactor-kafka spring-boot-2

Last synced: 10 Nov 2024

https://github.com/sergio11/iot_event_streaming_architecture

🌐📈 Unlock IoT potential with this Event Streaming Architecture: Apache Kafka and MQTT power real-time data handling. Visualize, analyze, and scale effortlessly. 🚀💻📊

cadvisor-exporter docker docker-compose grafana kafka kafka-connect kafka-streams micrometer mongodb node-exporter prometheus spring-boot spring-boot-actuator zookeeper

Last synced: 14 Dec 2024

https://github.com/influxdata/kafka-connect-influxdb

InfluxDB 2 Connector for Kafka

influxdb kafka kafka-connect

Last synced: 14 Nov 2024

https://github.com/90poe/connectctl

A kafka connect util

golang kafka-connect kafka-connector

Last synced: 12 Nov 2024

https://github.com/jcustenborder/kafka-connect-cdc-postgres

Kafka Connect connector for CDC data from postgres

cdc kafka-connect postgres

Last synced: 12 Oct 2024