{"id":13774894,"url":"https://github.com/Azure-Samples/azure-sql-db-change-stream-debezium","last_synced_at":"2025-05-11T07:30:44.966Z","repository":{"id":48035408,"uuid":"200424302","full_name":"Azure-Samples/azure-sql-db-change-stream-debezium","owner":"Azure-Samples","description":"SQL Server Change Stream sample using Debezium","archived":false,"fork":false,"pushed_at":"2023-10-23T18:36:12.000Z","size":1129,"stargazers_count":94,"open_issues_count":2,"forks_count":32,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-04-11T17:02:21.359Z","etag":null,"topics":["azure-functions","azure-sql","cdc","change-data-capture","change-feed","change-streams","debezium","event-hubs","kafka","sql-server"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azure-Samples.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-08-03T21:55:26.000Z","updated_at":"2024-02-12T12:28:06.000Z","dependencies_parsed_at":"2024-01-13T10:43:12.881Z","dependency_job_id":"a4f82b69-c56f-4161-ba0d-9a049687d111","html_url":"https://github.com/Azure-Samples/azure-sql-db-change-stream-debezium","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure-Samples%2Fazure-sql-db-change-stream-debezium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure-Samples%2Fazure-sql-db-change-stream-debezium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure-Samples%2Fazure-sql-db-change-stream-debezium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure-Samples%2Fazure-sql-db-change-stream-debezium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azure-Samples","download_url":"https://codeload.github.com/Azure-Samples/azure-sql-db-change-stream-debezium/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253533062,"owners_count":21923358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-functions","azure-sql","cdc","change-data-capture","change-feed","change-streams","debezium","event-hubs","kafka","sql-server"],"created_at":"2024-08-03T17:01:31.467Z","updated_at":"2025-05-11T07:30:44.537Z","avatar_url":"https://github.com/Azure-Samples.png","language":"C#","funding_links":[],"categories":["Code Samples"],"sub_categories":["Change Stream"],"readme":"---\npage_type: sample\nlanguages:\n- tsql\n- sql\n- csharp\nproducts:\n- azure-sql-database\n- sql-server\n- azure-sql-managed-instance\n- azure-sqlserver-vm\n- azure\n- dotnet\n- azure-functions\n- azure-event-hubs\ndescription: \"Create a Change Data Feed from Azure SQL or SQL Server using the Open Source tool Debezium\"\n---\n\n# Azure SQL / SQL Server Change Stream with Debezium\n\n\u003c!-- \nGuidelines on README format: https://review.docs.microsoft.com/help/onboard/admin/samples/concepts/readme-template?branch=master\n\nGuidance on onboarding samples to docs.microsoft.com/samples: https://review.docs.microsoft.com/help/onboard/admin/samples/process/onboarding?branch=master\n\nTaxonomies for products and languages: https://review.docs.microsoft.com/new-hope/information-architecture/metadata/taxonomies?branch=master\n--\u003e\n\n![License](https://img.shields.io/badge/license-MIT-green.svg)\n\nAzure SQL Database and SQL Server Change Stream sample using [Debezium](https://debezium.io/). A change feed or change stream allow applications to access real-time data changes, using standard technologies and well-known API, to create modern applications using the full power of database like SQL Server.\n\nDebezium make use of [Change Data Capture](https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-2017), so it can be used with SQL Server on-premises and the whole [Azure SQL](https://azure.microsoft.com/en-us/products/azure-sql) family (https://azure.microsoft.com/en-us/products/azure-sql) (Azure SQL MI, Azure SQL DB, SQL Server on VM).\n\nWith Debezium and Azure SQL / SQL Server you can not only create more modern and reactive applications that handle data changes in near real time with a minium impact on the database, but you can also use it to implement your Hybrid IT strategy, still using On-Prem SQL Server but relying on Azure for all your computing needs, taking advantage of PaaS offerings like EventHubs and Azure Functions. This sample will show how to do that.\n\n![SQL Server Change Stream](./documentation/sql-server-change-stream.gif)\n\nFor those who want more details, Debezium uses Apache Kafka, which is not natively available on Azure as a PaaS offer. But luckily EventHubs offer almost 100% support, so we can use it instead of Kafka and make maintenance and scalability easier.\n\n[Event Hubs for Kafka](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview)\n\n## Step by Step Guide\n\nThis step by step guide uses Wide World Importers sample database from here:\n\nhttps://github.com/Microsoft/sql-server-samples/releases/tag/wide-world-importers-v1.0\n\nMake sure you download the OLTP database if you want to follow this guide without having to change a thing. To restore the database on Azure SQL, you can use the scripts provided here: [Restore Database in Azure SQL](https://github.com/yorek/azure-sql-db-samples/tree/master/samples/01-restore-database).\n\n### Create Debezium User\n\nDebezium needs to query the database so a dedicated user is used throughout the sample. User has `db_owner` access to make the script simpler. In real world you may want to tighten security a bit more.\n\nTo create the login and user run the script `00-setup-database-user.sql` from the `sql` folder (pertinent to the service or product you are using, Azure SQL or SQL Server / Azure SQL MI) on the database where you have restored Wide World Importers database.\n\n### Enable Change Data Capture\n\nDebezium uses Change Data Capture to capture all the changes done to selected tables.\n\nIn this samples only two tables are monitored:\n\n- Sales.Orders\n- Warehouse.StockItems\n\nThe script `01-enable-cdc.sql` enable Change Data Capture on the aforementioned tables.\n\n### Create an Azure Event Hubs\n\nAll data gathered by Change Data Capture will be send to Event Hubs, so create an Azure Event Hubs in your Azure Subscription. Using the [Azure Cloud Shell](https://shell.azure.com/) Bash:\n\n```bash\n# Set vars\nRESOURCE_GROUP=debezium\nLOCATION=eastus\nEVENTHUB_NAMESPACE=debezium\nEVENTHUB_SCHEMA_HISTORY=schemahistory\n\n# create group\naz group create \\\n    --name $RESOURCE_GROUP \\\n    --location $LOCATION\n\n# create eventhub namespace with kafka enabled\naz eventhubs namespace create \\\n    --name $EVENTHUB_NAMESPACE \\\n    --resource-group $RESOURCE_GROUP \\\n    --location $LOCATION \\\n    --enable-kafka\n\n# create eventhub for schema history\naz eventhubs eventhub create \\\n    --resource-group $RESOURCE_GROUP \\\n    --namespace $EVENTHUB_NAMESPACE \\\n    --name $EVENTHUB_SCHEMA_HISTORY \\\n    --partition-count 1 \\\n    --cleanup-policy Delete \\\n    --retention-time-in-hours 168\n```\n\nLater in the configuration process you'll need the EventHubs connection string, so grab it and store it somewhere:\n\n```bash\nRESOURCE_GROUP=debezium\nEVENTHUB_NAMESPACE=debezium\n\naz eventhubs namespace authorization-rule keys list \\\n    --resource-group $RESOURCE_GROUP \\\n    --namespace-name $EVENTHUB_NAMESPACE \\\n    --name RootManageSharedAccessKey \\\n    --query \"primaryConnectionString\" \\\n    --output tsv\n ```\n\n### Run Debezium\n\n#### Pre-Requisites\n\nIn order to run Debezium you have to install and configure Apache Kafka, Apache Zookeper and Kafka Connect. If you already know how to do that, or your already have a testing or development environment, well, perfect. Go and install Debezium SQL Server Connector: [Installing a Debezium Connector](https://debezium.io/docs/install/stable/#installing_a_debezium_connector).\n\nIf prefer a more lean and quick easy to start using Debezium, you can just use the [Debezium Docker Image](https://github.com/debezium/docker-images), that provides anything you need to run a test instance of Debezium. Just make sure you have [Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose) installed. In the `debezium/on-prem` folder you can find all the scripts needed to run Debezium using Docker.\n\n#### Configure Environment\n\nDocker Compose will use `.env` to get the environment variables values used in the `.yaml` configuration file. The provided `.env.template` file look like the following:\n\n```bash\nDEBEZIUM_VERSION=2.7\nEVENTHUB_NAMESPACE=\u003ceventhub_namespace\u003e\nEVENTHUB_CONNECTION_STRING=\u003ceventhub_connection_string\u003e\n```\n\nCopy it and create a new `.env` file. Leave the version set to `2.7`. Change the `EVENTHUB_NAMESPACE` to the EventHubs name you created before. Also set `EVENTHUB_CONNECTION_STRING` to hold the EventHubs connection string you got before. Make sure not to use any additional quotes or double quotes.\n\n#### The .yaml file\n\nIf you are just interested in testing Debezium you can safely skip this section and move to the next one to start Debezium. If you want to understand how to make Debezium work with Event Hubs, read on.\n\nDebezium needs Apache Kafka to run, NOT Azure Event Hubs. Luckily for us, Azure Event Hubs exposes a Kafka-Compatible endpoint, so we can still enjoy Kafka with all the comfort of a PaaS offering. There are a few tweeks needed in order to make Debezium working with Azure Event Hubs.\n\nFirst of all EventHubs requires authentication. This part is taken care from the configuration settings that looks like the following:\n\n```yaml\n- *_SECURITY_PROTOCOL=SASL_SSL\n- *_SASL_MECHANISM=PLAIN\n- *_SASL_JAAS_CONFIG=[...]\n```\n\nDocumentation on EventHubs Kafka Authentication and Kafka Connect is available here:\n\n[Integrate Apache Kafka Connect support on Azure Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-connect-tutorial)\n\nSince we're running a Docker Image, we cannot really change the configuration file, but Debezium allows pass-through configurations:\n\n[Debezium Connect-Base](https://github.com/debezium/docker-images/tree/master/connect-base)\n\nThere is additional caveat to keep in mind. EventHubs security uses the string `$ConnectionString` as username. In order to avoid to have Docker Compose to treat it as a variable instead, a double dollar sign `$$` needs to be used:\n\n[Docker Compose Config File Variable Substitution](https://docs.docker.com/compose/compose-file/#variable-substitution)\n\nTwo other options useful for running Debezium on Azure Event Hubs are the following:\n\n```yaml\n- CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false\n- CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=true\n```\n\nThey control if the schema is sent with the data or not. Since the Azure Event Hubs only support values, as opposed to Apache Kafka, which everything is actually a key-value pair, the schema generation for the key section can be safely turned off. While this can also be done for the value part, it is not recommended as some data type are serialized in a Kafka-specific way and you need to know their \"sematic\" type in order to recreate the correct value.\n\n[Debezium SQL Server Connector Data Types](https://debezium.io/docs/connectors/sqlserver/#data-types)\n\nHere's a sample of a schema for a \"create\" (INSERT) event:\n\n[Debezium SQL Server Connector Create Event Sample](https://debezium.io/docs/connectors/sqlserver/#create-events)\n\n#### Start Debezium\n\nDebezium can now be started. If you're using the Docker Images you can just do this by running `debezium/on-prem/start-debezium.ps1` (or the `.sh` file if you're on Linux/WSL)\n\nOnce the startup has finished, you'll see something like\n\n```text\n[Worker clientId=connect-1, groupId=1] Finished starting connectors and tasks   [org.apache.kafka.connect.runtime.distributed.DistributedHerder]\n```\n\nyou will see four topics (or EventHub to use the Azure EventHubs nomenclature):\n\n```bash\n\nRESOURCE_GROUP=debezium\nEVENTHUB_NAMESPACE=debezium\n\naz eventhubs eventhub list \\\n    --resource-group $RESOURCE_GROUP \\\n    --namespace-name $EVENTHUB_NAMESPACE \\\n    --output table\n```\n\nand the result will show:\n\n- debezium_configs\n- debezium_offsets\n- debezium_statuses\n- schemahistory\n\nto explore Azure Event Hubs is strongly suggest to download and use [Service Bus Explorer](https://github.com/paolosalvatori/ServiceBusExplorer)\n\n#### Register SQL Server Connector\n\nNow that Debezium is running, the SQL Server Connector (which is used both for connecting to Azure SQL or SQL Server) can be registered. Before doing that, make sure to specify the correct connection for your SQL Server instance in a file named `sqlserver-connector-config.json`. You can create one using the template file [sqlserver-connector-config.json.template](debezium/sqlserver-connector-config.json.template) file.\n\nMake sure to change the following properties to match your SQL Server configuration:\n\n```json\n\"database.hostname\": \"\u003csql_server_name\u003e.database.windows.net\",\n\"database.names\": \"\u003cdb_name\u003e\",\n\"database.user\" : \"\u003cdebezium_user_name\u003e\",\n\"database.password\" : \"\u003cdebezium_user_password\u003e\",\n```\n\nYou would need to replace `\u003csql_server_name\u003e`, `\u003cdb_name\u003e`, `\u003cdebezium_user_name\u003e`, and `\u003cdebezium_user_password\u003e` with proper values.\n\nAlso, make sure to change the following properties to match your Azure Event Hub configuration:\n\n```json\n\"schema.history.internal.kafka.bootstrap.servers\": \"\u003ceventhub_namespace\u003e.servicebus.windows.net:9093\",\n\"schema.history.internal.kafka.topic\": \"\u003ceventhub_schema_history\u003e\",\n\"schema.history.internal.consumer.sasl.jaas.config\": \"org.apache.kafka.common.security.plain.PlainLoginModule required username=\\\"$ConnectionString\\\" password=\\\"\u003ceventhub_connectionstring\u003e\\\";\",\n\"schema.history.internal.producer.sasl.jaas.config\": \"org.apache.kafka.common.security.plain.PlainLoginModule required username=\\\"$ConnectionString\\\" password=\\\"\u003ceventhub_connectionstring\u003e\\\";\",\n```\n\nYou would need to replace `\u003ceventhub_namespace\u003e`, `\u003ceventhub_schema_history\u003e`, and `\u003ceventhub_connectionstring\u003e` with proper values.\n\nAll the other values used are explained in detail here:\n\n[SQL Server Connector Configuration Values](./documentation/SQL-Server-Connector-Configuration-Value.md)\n\nOnce the configuration file is set, just register that using `debezium/on-prem/register-connector.ps1`.\n\nDepending on how big your tables are, it make take a while (more on this later). Once you see the following message:\n\n```text\nSnapshot step 8 - Finalizing   [io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource]\n```\n\nand no other errors or exception before that, you'll know that the SQL Server Connector is correctly running.\n\n### Make sample changes\n\nNow that Debezium is running and fully configured, you can generate a new Sales Order and insert, update and delete some data in the Stock table. You can use the following scripts:\n\n```bash\n./sql/.../02-create-new-sales-order.sql\n./sql/.../03-modify-warehouse-stock.sql\n```\n\nAfter running the script you can use Service Bus Explorer or VS Code Event Hub Explorer to consume the stream of changes sent to Azure Event Hubs. You'll notice a new topic named `wwi`. That's where we instructed Debezium to send all the changes detected to the monitored tables.\n\n### Consume Change Stream using an Azure Functions\n\nOne way to quickly react to the Change Stream data coming from Debezium is to use Azure Functions. A sample is available in folder `azure-function`. The easiest way to run the sample is to open it from VS Code or via [Azure Function Core Tools](https://docs.microsoft.com/azure/azure-functions/functions-run-local), via `func start`. It will automatically recognize it as an Azure Function and download everything needed to run it.\n\nMake sure you have a `local.setting.json` that looks like the provided template. Copy the Azure Event Hubs connection string you got at the beginning into the `Debezium` configuration option.\n\nStart the function. As soon as the Azure Function runtime is running, the code will start to process the changes already available in EventHubs and you'll see something like this:\n\n```text\nEvent from Change Feed received:\n- Object: Sales.Orders\n- Operation: Insert\n- Captured At: 2019-08-04T22:35:59.0100000Z\n\u003e OrderID = 73625\n\u003e CustomerID = 941\n\u003e SalespersonPersonID = 3\n\u003e PickedByPersonID =\n\u003e ContactPersonID = 3141\n\u003e BackorderOrderID =\n\u003e OrderDate = 8/4/2019 12:00:00 AM\n\u003e ExpectedDeliveryDate = 8/5/2019 12:00:00 AM\n\u003e CustomerPurchaseOrderNumber = 4923\n\u003e IsUndersupplyBackordered = False\n\u003e Comments = Auto-generated\n\u003e DeliveryInstructions = Unit 17, 1466 Deilami Road\n\u003e InternalComments =\n\u003e PickingCompletedWhen =\n\u003e LastEditedBy = 3\n\u003e LastEditedWhen = 8/4/2019 10:35:58 PM\nExecuted 'ProcessDebeziumPayload' (Succeeded, Id=ee9d1080-64ff-4039-83af-69c4b12fa85f)\n```\n\n### Done\n\nCongratulations, you now have a working Change Stream from SQL Server. This opens up a whole new set of possibilities! Have fun!\n\n## Notes\n\n### Running Debezium on Azure\n\nIf you're using Debezium with Azure SQL MI or Azure SQL DB, you may want to run Debezium on Azure. Sample script to run the Debezium container on Azure Container Instances are available in the `debezium/azure` folder.\n\n### Connector Configuration\n\nMore details on SQL Server and Event Hubs specific configuration here:\n\n[SQL Server Connector Configuration Values](./documentation/SQL-Server-Connector-Configuration-Value.md)\n\n## Troubleshooting\n\n### Local Azure Function not working\n\nIf, after starting the Azure Function via `func start` you get the following error:\n\n```\nThe listener for function 'ProcessDebeziumPayload' was unable to start. Microsoft.WindowsAzure.Storage: No connection could be made because the target machine actively refused it. System.Net.Http: No connection could be made because the target machine actively refused it. System.Private.CoreLib: No connection could be made because the target machine actively refused it.\n```\n\nyou need to make sure that [Azure Storage Emulator](https://docs.microsoft.com/azure/storage/common/storage-use-azurite) is installed and started.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAzure-Samples%2Fazure-sql-db-change-stream-debezium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAzure-Samples%2Fazure-sql-db-change-stream-debezium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAzure-Samples%2Fazure-sql-db-change-stream-debezium/lists"}