{"id":22357139,"url":"https://github.com/gordonmurray/debezium_warpstream_http_sink","last_synced_at":"2026-02-27T13:10:47.496Z","repository":{"id":216958733,"uuid":"742588340","full_name":"gordonmurray/debezium_warpstream_http_sink","owner":"gordonmurray","description":"Using Debezium to send Change Data Capture (CDC) events to HTTP endpoints","archived":false,"fork":false,"pushed_at":"2024-11-29T21:59:45.000Z","size":8872,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-04T10:35:09.169Z","etag":null,"topics":["debezium","http","kafka","ngrok","warpstream"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gordonmurray.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2024-01-12T20:14:36.000Z","updated_at":"2024-11-29T21:59:48.000Z","dependencies_parsed_at":"2024-01-13T22:17:09.829Z","dependency_job_id":"6feeae6d-f4f1-4782-a146-10b6b2e9fae7","html_url":"https://github.com/gordonmurray/debezium_warpstream_http_sink","commit_stats":null,"previous_names":["gordonmurray/debezium_warpstream_http_sink"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gordonmurray/debezium_warpstream_http_sink","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fdebezium_warpstream_http_sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fdebezium_warpstream_http_sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fdebezium_warpstream_http_sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fdebezium_warpstream_http_sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gordonmurray","download_url":"https://codeload.github.com/gordonmurray/debezium_warpstream_http_sink/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fdebezium_warpstream_http_sink/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29896315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T12:09:13.686Z","status":"ssl_error","status_checked_at":"2026-02-27T12:09:13.282Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["debezium","http","kafka","ngrok","warpstream"],"created_at":"2024-12-04T14:13:19.044Z","updated_at":"2026-02-27T13:10:47.476Z","avatar_url":"https://github.com/gordonmurray.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Debezium, Warpstream and a HTTP sink\n\nUsing Debezium to send Change Data Capture (CDC) events to HTTP endpoints\n\nI wanted to try out the HTTP sink connector to see how sending data out to external endpoints might work.\n\nI created a small project to source data from a relational database using Debezium to populate some kafka topics and then added a connector to send data from one of the topics to a HTTP endpoint.\n\nIt works well and had a couple of pleasant surprises such as the connector auto creats topics in the kafka cluster to store a record of errors and successes of the HTTP calls. That's great for debugging issues. You can control the topic names in the connector config too.\n\nI created a docker compose file that creates a Mariadb Database with a tiny products table and a Debezium container. A first connector is added to source the data from the database and populate kafka topics. A second connector is then added to send events form one of the topics to a HTTP endpoint.\n\nI used Warpstream as my kafka cluster and I used Ngrok as a temporary endpoint to send data to so I could inspect the data is received.\n\nThe source connector to read data from a table in the database is located at files/connector_mariadb.json looks like the following:\n\n```json5\n{\n    \"connector.class\": \"io.debezium.connector.mysql.MySqlConnector\",\n\n    \"database.history.kafka.bootstrap.servers\": \"api-xxxxxxxxxx.warpstream.com:9092\",\n    \"database.history.kafka.topic\": \"history\",\n\n    // database connection details\n    \"database.hostname\": \"mariadb\",\n    \"database.password\": \"rootpassword\",\n    \"database.port\": \"3306\",\n    \"database.server.id\": \"12\",\n    \"database.server.name\": \"myconnector\",\n    \"database.user\": \"root\",\n    \"database.whitelist\": \"mydatabase\",\n\n    \"schema.history.internal.kafka.bootstrap.servers\": \"api-xxxxxxxxxx.warpstream.com:9092\",\n    \"schema.history.internal.kafka.topic\": \"schema-changes.mydatabase\",\n    \"table.whitelist\": \"mydatabase.products\",\n    \"tasks.max\": \"1\",\n    \"topic.prefix\": \"testing\",\n    \"transforms.unwrap.type\": \"io.debezium.transforms.ExtractNewRecordState\",\n    \"transforms\": \"unwrap\"\n}\n```\n\nThe fields to pay attention to in the JSON config are:\n\n* database.history.kafka.bootstrap.servers \u0026 chema.history.internal.kafka.bootstrap.servers - your kafka brokers, in my case Im using WarpStream\n* database.* fields - to specify the connection details to the database\n* table.whitelist - the table you want to pull data from, written in the format of schema.tablename\n\nDebezium will create a number of topics in the kafka cluster, in this case it will create a topic called testing.mydatabase.products bases on the prefix and table.whitelist contents.\n\n![Warpstream UI showing topics]({{ site.url }}/images/warpstream_topics.png)\n\nOnce the topic is in place with some records, I can then add the HTTP sink connector, located at files/connector_http.json. The connector config looks like the following:\n\n```json5\n{\n    \"connector.class\": \"io.confluent.connect.http.HttpSinkConnector\",\n    \"tasks.max\": \"1\",\n    \"topics\": \"testing.mydatabase.products\",\n    \"http.api.url\": \"https://xxxxxxxx.ngrok-free.app\",\n    \"request.method\": \"POST\",\n    \"headers\": \"Content-Type:application/json\",\n    \"batch.size\": \"3\",\n    \"max.retries\": \"3\",\n    \"retry.backoff.ms\": \"3000\",\n    \"connection.timeout.ms\": \"2000\",\n    \"request.timeout.ms\": \"5000\",\n\n    // add your warpstream endpoint here\n    \"confluent.topic.bootstrap.servers\": \"api-xxxxxxxxx.warpstream.com:9092\",\n    \"reporter.bootstrap.servers\": \"api-xxxxxxxxx.warpstream.com:9092\",\n\n    \"reporter.error.topic.name\": \"error-topic\",\n    \"reporter.error.topic.replication.factor\": \"1\",\n    \"reporter.result.topic.name\": \"success-topic\",\n    \"reporter.result.topic.replication.factor\": \"1\"\n}\n```\n\nThe fields to pay attention to in the JSON config are:\n\n* topics - thats the topic in kafka you want to respond to, in my case its 'products'\n* http.api.url - thats the URL to send the events to, in my case is an ngrok endpoint for testing\n* confluent.topic.bootstrap.servers \u0026 reporter.bootstrap.servers - your kafka brokers, in my case Im using WarpStream\n\n\nFor an API endpoint I used [ngrok](https://ngrok.com/), I started it locally using the following command and it gave me a URL to add to my HTTP connector\n\n```\nngrok http 80\n```\n\nOnce the connectors are updated with any changes you want to make you can start the containers.\n\nRun `docker-compose up -d` and connect to the Debezium container:\n\n```\ndocker exec -it debezium /bin/bash\n```\n\nDocker compose will upload the connectors in to the Debezium container, you can start the connectors using to following commands:\n\n\nCreate the source connector:\n\n```\ncurl -X PUT http://localhost:8083/connectors/my_database/config -H \"Content-Type: application/json\" -d @connector_mariadb.json\n```\n\n\nCreate the HTTP sink connector:\n\n```\ncurl -X PUT http://localhost:8083/connectors/http/config -H \"Content-Type: application/json\" -d @connector_http.json\n```\n\nYou should see at least 3 calls to the API endpoint, one for each of the 3 records thats in the database by default. Adding or updating records in the table will trigger more calls to the endpoint.\n\nThe events look like the following in Ngrok\n\n```\nStruct{id=3,name=Product C,price=39.99}\n```\n\nIt doesn't trigger a HTTP event for any schema changes, but any events afterwards will contain the data for any new columns that are added.\n\n```\nStruct{id=5,name=product E,price=33.00,notes=great product}\n```\n\nSchema changes are recorded in another topic called schema-changes.mydatabase so another HTTP sink could be used to monitor that topic if you wanted to watch for schema changes.\n\n### Useful commands\n\nList Connectors:\n\n```\ncurl localhost:8083/connectors\n```\n\nCheck Connector status:\n\n```\ncurl localhost:8083/connectors/my_database/status\n```\n\nUse the Kafka CLI to list topics in WarpStream\n\n```\n./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list\n```\n\nList the messages in a topic\n\n```\n./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testing.mydatabase.products --from-beginning\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Fdebezium_warpstream_http_sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgordonmurray%2Fdebezium_warpstream_http_sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Fdebezium_warpstream_http_sink/lists"}