{"id":23338333,"url":"https://github.com/abhirockzz/kafka-data-explorer-cdc","last_synced_at":"2025-10-28T17:31:13.854Z","repository":{"id":50071182,"uuid":"302124679","full_name":"abhirockzz/kafka-data-explorer-cdc","owner":"abhirockzz","description":null,"archived":false,"fork":false,"pushed_at":"2021-06-05T00:02:48.000Z","size":202,"stargazers_count":3,"open_issues_count":6,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-21T03:14:07.385Z","etag":null,"topics":["azure","azure-data-explorer","debezium","docker","go","kafka","kafka-connect","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhirockzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-07T18:24:09.000Z","updated_at":"2023-11-30T21:07:40.000Z","dependencies_parsed_at":"2022-09-19T05:40:37.810Z","dependency_job_id":null,"html_url":"https://github.com/abhirockzz/kafka-data-explorer-cdc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhirockzz%2Fkafka-data-explorer-cdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhirockzz%2Fkafka-data-explorer-cdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhirockzz%2Fkafka-data-explorer-cdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhirockzz%2Fkafka-data-explorer-cdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhirockzz","download_url":"https://codeload.github.com/abhirockzz/kafka-data-explorer-cdc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238688115,"owners_count":19513789,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-data-explorer","debezium","docker","go","kafka","kafka-connect","postgresql"],"created_at":"2024-12-21T03:13:37.729Z","updated_at":"2025-10-28T17:31:13.384Z","avatar_url":"https://github.com/abhirockzz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Change Data Capture from PostgreSQL to Azure Data Explorer using Kafka Connect\n\nThis quickstart demonstrates how you can use Change Data Capture to stream database modifications from PostgreSQL to [Azure Data Explorer](https://docs.microsoft.com/en-us/azure/data-explorer/) via [Apache Kafka](https://kafka.apache.org/). It revolves around [Kafka Connect](https://kafka.apache.org/documentation/#connect) which is a platform to build scalable and reliable streaming data pipelines. All the components run in Docker containers - this includes Kafka, Zookeeper, Kafka Connect workers and the event generator application.\n\n## Overview\n\n*Change Data Capture (CDC)* is a technique used to track row-level changes in database tables in response to create, update and delete operations. This is a powerful capability, but useful only if there is a way to tap into these event logs and make it available to other services which depend on that information.\n\n[Debezium](https://debezium.io/) is a distributed platform that builds on top of Change Data Capture features available in different databases. It provides a set of [Kafka Connect connectors](https://debezium.io/documentation/reference/1.2/connectors/index.html) which tap into row-level changes (using CDC) in database table(s) and convert them into event streams that are sent to Apache Kafka. Once the change log events are in Kafka, they will be available to all the downstream applications.\n\nHere is a high-level overview of the scenario. Please note that it has been kept simplified for demonstration purposes.\n\nData related to `Orders` is stored in a PostgreSQL database and contains information such as order ID, customer ID, city, transaction amount etc. Sample order data is continuously generated with the help of a custom Go application. This data is then picked up by a Kafka Connect source connector and sent to a Kafka topic. Once the data is in Kafka, another (Sink) connector sends them to Azure Data Explorer allow or further querying and analysis.\n\nHere is a gist of the individual components:\n\n**The foundation**\n\n- Apache Kafka - distributed streaming platform\n- Kafka Connect - tool for streaming data pipelines driven by source and sink connectors.\n\n**Source and destinations**\n\n- PostgreSQL: RDBMS to store `orders` data\n- Azure Data Explorer: fully managed Big Data Analytics platform\n\n**Kafka Connect connectors**\n\n- [Debezium connector for PostgreSQL](https://debezium.io/documentation/reference/1.2/connectors/postgresql.html) - Captures row-level changes a PostgreSQL database and sends them to Kafka\n- [Azure Data Explorer sink connector](https://github.com/Azure/kafka-sink-azure-kusto) - Open source connector that pushes data from Kafka into Azure Data Explorer\n\n## Pre-requisites\n\n- You will need a [Microsoft Azure account](https://docs.microsoft.com/azure/). Maybe try a [free one?](https://azure.microsoft.com/free/)\n- Install [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest) if you don't have it already (should be quick!) or just use the [Azure Cloud Shell](https://azure.microsoft.com/features/cloud-shell/) from your browser.\n- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install) installed\n- Clone this repo\n\n```shell\ngit clone https://github.com/abhirockzz/kafka-data-explorer-cdc\ncd kafka-data-explorer-cdc\n```\n\n## Docker Compose services\n\nAll the components in this demo run as Docker containers (except Azure Data Explorer of course) using Docker Compose. You can refer to the [docker-compose file in the repo](docker-compose.yaml): here is a quick overview:\n\nThe [debezium](https://hub.docker.com/r/debezium/kafka/) images are used for Kafka and Zookeeper\n\n```yaml\n  zookeeper:\n    image: debezium/zookeeper:1.2\n    ports:\n      - 2181:2181\n  kafka:\n    image: debezium/kafka:1.2\n    ports:\n      - 9092:9092\n    links:\n      - zookeeper\n    depends_on:\n      - zookeeper\n    environment:\n      - ZOOKEEPER_CONNECT=zookeeper:2181\n      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092\n```\n\n[debezium/example-postgres](https://hub.docker.com/r/debezium/example-postgres) is a handy Docker image for PostgreSQL and the `orders-gen` service is a custom Go application to seed random orders data into PostgreSQL\n\n```yaml\n  postgres:\n    image: debezium/example-postgres\n    ports:\n      - \"5432:5432\"\n    environment:\n      - POSTGRES_USER=postgres\n      - POSTGRES_PASSWORD=postgres\n  orders-gen:\n    build:\n      context: ./orders-generator\n    links:\n      - postgres\n    depends_on:\n      - postgres\n```\n\nKafka Connect workers: Both the connectors (source and sink) run within this container. The PostgreSQL connector is built into [debezium/connect](https://hub.docker.com/r/debezium/connect) image and the Azure Data Explorer connector is installed on top of that (you can check this [Dockerfile](connector/Dockerfile))\n\n```yaml\n  connect:\n    build:\n      context: ./connector\n      args:\n        KUSTO_KAFKA_SINK_VERSION: 1.0.1\n    ports:\n      - 8083:8083\n    links:\n      - kafka\n      - postgres\n    depends_on:\n      - kafka\n      - postgres\n      - orders-gen\n    environment:\n      - BOOTSTRAP_SERVERS=kafka:9092\n      - GROUP_ID=adx\n      - CONFIG_STORAGE_TOPIC=my_connect_configs\n      - OFFSET_STORAGE_TOPIC=my_connect_offsets\n      - STATUS_STORAGE_TOPIC=my_connect_statuses\n```\n\nBefore we dive into things, let's make sure you have setup and configured Azure Data Explorer\n\n## Setup and configure Azure Data Explorer\n\n1. Create an Azure Data Explorer cluster and a database - you can use this [handy quickstart from the documentation](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal)\n\n2. Create a table (`Orders`) and the mapping (`OrdersEventMapping`) using the [KQL](https://docs.microsoft.com/en-us/azure/data-explorer/kql-quick-reference) below:\n\n```kusto\n.create table Orders (orderid: string, custid: string, city: string, amount: int)\n\n.create table Orders ingestion json mapping 'OrdersEventMapping' '[{\"column\":\"orderid\",\"Properties\":{\"path\":\"$.orderid\"}},{\"column\":\"custid\",\"Properties\":{\"path\":\"$.custid\"}},{\"column\":\"city\",\"Properties\":{\"path\":\"$.city\"}},{\"column\":\"amount\",\"Properties\":{\"path\":\"$.amount\"}}]'\n```\n\n(Optional) For the purposes of this demo, update the ingestion batching policy. This will allow records to be ingested quickly:\n\n```kusto\n.alter table Orders policy ingestionbatching @'{\"MaximumBatchingTimeSpan\":\"00:00:30\", \"MaximumNumberOfItems\": 500, \"MaximumRawDataSizeMB\": 1024}'\n\n.show table \u003center database name\u003e.Orders policy ingestionbatching\n```\n\n3. Create a Service Principal in order for the connector to authenticate and connect to Azure Data Explorer service.\n\nUse `az ad sp create-for-rbac` command:\n\n```\naz ad sp create-for-rbac -n \"kusto-sp\"\n```\n\nYou will get a JSON response as such - please note down the `appId`, `password` and `tenant` as you will be using them in subsequent steps\n\n```json\n{\n  \"appId\": \"fe7280c7-5705-4789-b17f-71a472340429\",\n  \"displayName\": \"kusto-sp\",\n  \"name\": \"http://kusto-sp\",\n  \"password\": \"29c719dd-f2b3-46de-b71c-4004fb6116ee\",\n  \"tenant\": \"42f988bf-86f1-42af-91ab-2d7cd011db42\"\n}\n```\n\n4. Add permissions to your database\n\nProvide appropriate role to the Service principal you just created. To assign the `admin` role, [follow this guide](https://docs.microsoft.com/en-us/azure/data-explorer/manage-database-permissions#manage-permissions-in-the-azure-portal) to use the Azure portal or use the following command in your Data Explorer cluster\n\n```kusto\n.add database \u003cdatabase name\u003e admins  ('aadapp=\u003cservice principal AppID\u003e;\u003cservice prinicpal TenantID\u003e') 'AAD App'\n```\n\nLet's dive into some hands-on stuff. In the next few sections, you will:\n\n- Get the environment up and running (all the containers)\n- Setup the source (PostgreSQL) and sink (Azure Data Explorer) connectors\n- Validate the end to end pipeline\n\n## Start the Docker containers\n\nSince this is based on Docker Compose, starting up the environment is very convenient - just a single command:\n\n```shell\ndocker-compose --project-name kusto-kafka-cdc up --build\n```\n\nThis will start Kafka, Zookeeper, Kafka Connect workers, PostgreSQL and a custom events generator application that will start inserting random order events to the `orders_info` table in PostgreSQL. \n\n\u003e It might take a while to download and start the containers - note that this is just a one time process.\n\nOnce the containers have started, to confirm, you can run:\n\n```shell\ndocker-compose -p kusto-kafka-cdc ps\n\n            Name                          Command              State                          Ports                       \n--------------------------------------------------------------------------------------------------------------------------\nkusto-kafka-cdc_connect_1      /docker-entrypoint.sh start     Up      0.0.0.0:8083-\u003e8083/tcp, 8778/tcp, 9092/tcp,        \n                                                                       9779/tcp                                           \nkusto-kafka-cdc_kafka_1        /docker-entrypoint.sh start     Up      8778/tcp, 0.0.0.0:9092-\u003e9092/tcp, 9779/tcp         \nkusto-kafka-cdc_orders-gen_1   /orders-gen                     Up                                                         \nkusto-kafka-cdc_postgres_1     docker-entrypoint.sh postgres   Up      0.0.0.0:5432-\u003e5432/tcp                             \nkusto-kafka-cdc_zookeeper_1    /docker-entrypoint.sh start     Up      0.0.0.0:2181-\u003e2181/tcp, 2888/tcp, 3888/tcp,        \n                                                                       8778/tcp, 9779/tcp\n```\n\nAt this point you can also do quick sanity check to confirm that the order information is being persisted. Use a tool of your choice to connect to the PostgreSQL instance (I have used [psql](https://www.postgresql.org/docs/13/app-psql.html) in the example below).\n\n**Enter *postgres* when prompted for a password**\n\n```sql\npsql -h localhost -p 5432 -U postgres -W -d postgres\n\nselect * from inventory.orders_info order by orderid desc limit 5;\n\n orderid | custid | amount | city  \n---------+--------+--------+-------\n      48 |    840 |    140 | city3\n      47 |    613 |    132 | city3\n      46 |    904 |    124 | city3\n      45 |    529 |    118 | city1\n      44 |    711 |    187 | city2\n(5 rows)\n```\n\n\u003e This will give you the five most recent orders.\n\nTo stream order information to Kafka, we need to configure and start an instance of the Debezium PostgreSQL source connector.\n\n## Debezium PostgreSQL source connector setup\n\nCopy the JSON contents below to a file (you can name it `pg-source-config.json`).\n\n```json\n{\n    \"name\": \"pg-orders-source\",\n    \"config\": {\n        \"connector.class\": \"io.debezium.connector.postgresql.PostgresConnector\",\n        \"database.hostname\": \"postgres\",\n        \"database.port\": \"5432\",\n        \"database.user\": \"postgres\",\n        \"database.password\": \"postgres\",\n        \"database.dbname\": \"postgres\",\n        \"database.server.name\": \"myserver\",\n        \"plugin.name\": \"wal2json\",\n        \"table.whitelist\": \"inventory.orders_info\",\n        \"value.converter\": \"org.apache.kafka.connect.json.JsonConverter\"\n    }\n}\n```\n\nTo start the connector, simply use the Kafka Connect REST endpoint to submit the configuration.\n\n```shell\ncurl -X POST -H \"Content-Type: application/json\" --data @pg-source-config.json http://localhost:8083/connectors\n\n# to confirm\ncurl http://localhost:8083/connectors/pg-orders-source\n```\n\nLet's peek into the Kafka topic and take a look at the change data capture events produced by the source connector.\n\n```shell\ndocker exec -it kusto-kafka-cdc_kafka_1 bash\n```\n\nYou will be dropped into a shell (inside the container):\n\n```\ncd bin \u0026\u0026 ./kafka-console-consumer.sh --topic myserver.inventory.orders_info --bootstrap-server kafka:9092 --from-beginning\n```\n\nEach event in topic is corresponding to a specific order. It is in a JSON format that looks like what's depicted below. Please note that the payload also contains the entire schema which has been removed for brevity.\n\n```json\n{\n    \"schema\": {....},\n    \"payload\": {\n        \"before\": null,\n        \"after\": {\n            \"orderid\": 51,\n            \"custid\": 306,\n            \"amount\": 183,\n            \"city\": \"city1\"\n        },\n        \"source\": {\n            \"version\": \"1.2.1.Final\",\n            \"connector\": \"postgresql\",\n            \"name\": \"myserver\",\n            \"ts_ms\": 1602057392691,\n            \"snapshot\": \"false\",\n            \"db\": \"postgres\",\n            \"schema\": \"inventory\",\n            \"table\": \"orders_info\",\n            \"txId\": 653,\n            \"lsn\": 34220200,\n            \"xmin\": null\n        },\n        \"op\": \"c\",\n        \"ts_ms\": 1602057392818,\n        \"transaction\": null\n    }\n}\n```\n\n## Azure Data Explorer sink connector setup\n\nWe have the change-data flowing into Kafka from PostgreSQL. It's time to bring that into Azure Data Explorer using its Kafka sink connector.\n\nCopy the JSON contents below to a file (you can name it `adx-sink-config.json`). Replace the values for the following attributes as per your Azure Data Explorer setup - `aad.auth.authority`, `aad.auth.appid`, `aad.auth.appkey`, `kusto.tables.topics.mapping` (the database name) and `kusto.url`\n\n```json\n{\n    \"name\": \"adx-orders-sink\",\n    \"config\": {\n        \"connector.class\": \"com.microsoft.azure.kusto.kafka.connect.sink.KustoSinkConnector\",\n        \"flush.size.bytes\": 10000,\n        \"flush.interval.ms\": 30000,\n        \"tasks.max\": 2,\n        \"topics\": \"myserver.inventory.orders_info\",\n        \"kusto.tables.topics.mapping\": \"[{'topic': 'myserver.inventory.orders_info','db': '\u003center database name\u003e', 'table': 'Orders','format': 'json', 'mapping':'OrdersEventMapping'}]\",\n        \"aad.auth.authority\": \"\u003center tenant ID from service principal info\u003e\",\n        \"kusto.url\": \"https://ingest-\u003center cluster name\u003e.\u003center region\u003e.kusto.windows.net\",\n        \"aad.auth.appid\": \"\u003center app ID from service principal info\u003e\",\n        \"aad.auth.appkey\": \"\u003center password from service principal info\u003e\",\n        \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n        \"transforms\": \"unwrap\",\n        \"transforms.unwrap.type\": \"io.debezium.transforms.ExtractNewRecordState\"\n    }\n}\n```\n\nNotice the part in the config where we used Kafka Connect [Single Message Transformation](https://kafka.apache.org/documentation/#connect_transforms) (SMT) - specifically this is the `ExtractNewRecordState` that debezium provides. You can read up on it [in the documentation](https://debezium.io/documentation/reference/1.2/configuration/event-flattening.html)\n\n```json\n\"transforms\": \"unwrap\",\n\"transforms.unwrap.type\": \"io.debezium.transforms.ExtractNewRecordState\"\n```\n\nIt removes the schema from the JSON payload and strips down the data to only what's required. In this case, all we are looking for the order info from the `after` attribute (in the payload). For e.g.\n\n```json\n{\n    \"orderid\": 51,\n    \"custid\": 306,\n    \"amount\": 183,\n    \"city\": \"city1\"\n}\n```\n\nThus, only the relevant data sent to Azure Data Explorer. Also the Kafka topic contains the entire change data event (along with the schema) which can be leveraged by any downstream service!\n\nTo install the connector, just use the Kafka Connect REST endpoint like before:\n\n```shell\ncurl -X POST -H \"Content-Type: application/json\" --data @adx-sink-config.json http://localhost:8083/connectors\n\n# check status\ncurl http://localhost:8083/connectors/adx-orders-sink/status\n```\n\nThe connector should spin into action, authenticate to Azure Data Explorer and start batching ingestion processes.\n\n\u003e Note that `flush.size.bytes` and `flush.interval.ms` are used to regulate the batching process. Please refer to the [connector documentation](https://github.com/Azure/kafka-sink-azure-kusto/blob/master/README.md#5-sink-properties) for details on the individual properties.\n\nSince the flush configuration for the connector and the batching policy for the `Orders` table in Azure Data Explorer is pretty aggressive (for demonstration purposes), you should see data flowing into Data Explorer quickly.\n\n## Query Data Explorer\n\nYou can query the Orders table in Data Explorer to slice and dice the data. Here are a few simple queries you could try.\n\nFind out the average sales per city and represent that as a column chart:\n\n```kusto\nOrders\n| summarize avg_sales = avg(amount) by city \n| render columnchart\n```\n\n![](images/column-chart.png)\n\nThe total purchase amount per city, represented as a pie chart:\n\n```kusto\nOrders \n| summarize total = sum(amount) by city \n| sort by total\n| render piechart \n```\n\n![](images/pie-chart.png)\n\n\nNumber of orders per city, represented as a line chart:\n\n```kusto\nOrders\n| summarize orders = count() by city\n| sort by orders\n| render linechart   \n```\n\n![](images/line-chart.png)\n\n## Clean up\n\nTo stop the containers, you can:\n\n```shell\ndocker-compose -p kusto-kafka-cdc down -v\n```\n\nTo delete the Azure Data Explorer cluster/database, use [az cluster delete](https://docs.microsoft.com/cli/azure/kusto/cluster?view=azure-cli-latest#az-kusto-cluster-delete) or [az kusto database delete](https://docs.microsoft.com/cli/azure/kusto/database?view=azure-cli-latest#az-kusto-database-delete)\n\n```azurecli\naz kusto cluster delete -n \u003ccluster name\u003e -g \u003cresource group name\u003e\naz kusto database delete -n \u003cdatabase name\u003e --cluster-name \u003ccluster name\u003e -g \u003cresource group name\u003e\n```\n\n## Other resources\n\n- [Data Ingestion overview](https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-overview)\n- Explore [what you can do with KQL](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/samples)\n- [Visualize data with Azure Data Explorer dashboards](https://docs.microsoft.com/en-us/azure/data-explorer/azure-data-explorer-dashboards)\n- [Data Explorer connector features](https://github.com/Azure/kafka-sink-azure-kusto/blob/master/README.md#3-features-supported)\n- [Debezium connectors](https://debezium.io/documentation/reference/1.2/connectors/index.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhirockzz%2Fkafka-data-explorer-cdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhirockzz%2Fkafka-data-explorer-cdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhirockzz%2Fkafka-data-explorer-cdc/lists"}