{"id":19052974,"url":"https://github.com/kayvansol/kafka-source","last_synced_at":"2025-09-07T06:33:13.739Z","repository":{"id":237331881,"uuid":"794311398","full_name":"kayvansol/kafka-source","owner":"kayvansol","description":"Debezium Source Connector from SQL Server to Apache Kafka","archived":false,"fork":false,"pushed_at":"2024-05-04T18:09:11.000Z","size":557,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-08T23:33:00.356Z","etag":null,"topics":["akhq","arvancloud","cdc","confluent-kafka","confluentinc","debezium-sqlserver","docker-compose","kafka-connect","sql","sqlserver","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"TSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kayvansol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-30T21:50:59.000Z","updated_at":"2024-08-08T13:21:13.000Z","dependencies_parsed_at":"2024-05-04T19:24:16.667Z","dependency_job_id":null,"html_url":"https://github.com/kayvansol/kafka-source","commit_stats":null,"previous_names":["kayvansol/kafka-source"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Fkafka-source","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Fkafka-source/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Fkafka-source/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayvansol%2Fkafka-source/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kayvansol","download_url":"https://codeload.github.com/kayvansol/kafka-source/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232186524,"owners_count":18485378,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akhq","arvancloud","cdc","confluent-kafka","confluentinc","debezium-sqlserver","docker-compose","kafka-connect","sql","sqlserver","zookeeper"],"created_at":"2024-11-08T23:28:37.502Z","updated_at":"2025-01-02T10:42:20.956Z","avatar_url":"https://github.com/kayvansol.png","language":"TSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Debezium source connector from SQL Server to Apache Kafka\n\n![alt text](https://github.com/kayvansol/kafka-source/blob/main/img/debezum.png?raw=true)\n\nDocker file for the connector that installing debezium-connector-sqlserver library [Dockerfile](https://github.com/kayvansol/kafka-source/blob/main/Dockerfile):\n```bash\nFROM docker.arvancloud.ir/confluentinc/cp-kafka-connect:latest\n\nRUN confluent-hub install confluentinc/kafka-connect-jdbc:10.7.6 --no-prompt\nRUN confluent-hub install debezium/debezium-connector-sqlserver:2.4.2 --no-prompt\n```\n\nthe build command :\n```bash\ndocker build -t docker.arvancloud.ir/confluentinc/cp-kafka-connect:latest .\n```\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/kafka-connect-with_debezum.png?raw=true)\n\n```bash\ncd kafka-source\n```\nDocker Compose file [docker-compose.yml](https://github.com/kayvansol/kafka-source/blob/main/docker-compose.yml) :\n```yml\n---\nversion: '3'\n\nservices:\n\n  zookeeper2:\n    image: docker.arvancloud.ir/confluentinc/cp-zookeeper:latest\n    hostname: zookeeper2\n    container_name: zookeeper2\n    ports:\n      - \"2181:2181\"\n    environment:\n      ZOOKEEPER_CLIENT_PORT: 2181\n      ZOOKEEPER_TICK_TIME: 2000\n    networks:\n      - kafka-network1\n\n  kafka1:\n    image: docker.arvancloud.ir/confluentinc/cp-kafka:latest\n    hostname: kafka1\n    container_name: kafka1\n    depends_on:\n      - zookeeper2\n    ports:\n      - \"29092:29092\"\n      - \"9092:9092\"\n    environment:\n      KAFKA_BROKER_ID: 1\n      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper2:2181'\n      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\n      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092\n      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0\n      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1\n      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1\n      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'\n      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka1:29092\n      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper2:2181\n      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1\n      CONFLUENT_METRICS_ENABLE: 'false'\n    networks:\n      - kafka-network1\n\n  connect1:\n    image: docker.arvancloud.ir/confluentinc/cp-kafka-connect:latest\n    hostname: connect1\n    container_name: connect1\n    depends_on:\n      - zookeeper2\n      - kafka1\n    ports:\n      - '8083:8083'\n    environment:\n      CONNECT_BOOTSTRAP_SERVERS: 'kafka1:29092'\n      CONNECT_REST_ADVERTISED_HOST_NAME: connect1\n      CONNECT_REST_PORT: 8083\n      CONNECT_GROUP_ID: compose-connect-group\n      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs\n      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1\n      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000\n      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets\n      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1\n      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status\n      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1\n      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter\n      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter\n      CONNECT_INTERNAL_KEY_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'\n      CONNECT_INTERNAL_VALUE_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'\n      CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE: 'false'\n      CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: 'false'\n      CONNECT_ZOOKEEPER_CONNECT: 'zookeeper2:2181'\n      # CLASSPATH required due to CC-2422\n      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-5.4.1.jar\n      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: 'io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor'\n      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: 'io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor'\n      CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components'\n      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR\n    networks:\n      - kafka-network1\n\n  akhq1:\n    image: tchiotludo/akhq\n    hostname: web-ui1\n    container_name: web-ui1\n    environment:\n      AKHQ_CONFIGURATION: |\n        akhq:\n          connections:\n            docker-kafka-server:\n              properties:\n                bootstrap.servers: \"kafka1:29092\"              \n              connect:\n                - name: \"connect\"\n                  url: \"http://connect1:8083\"\n#             ksqldb:\n#               - name: \"ksqldb\"\n#                 url: \"http://ksqldb-server:8088\"\n\n    ports:\n      - 8080:8080\n    links:\n      - kafka1\n    networks:\n      - kafka-network1\n\nnetworks:\n  kafka-network1:\n    driver: bridge\n    name: kafka-network1\n\n```\n```\ndocker compose up\n```\nDocker Desktop :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/Containers.png?raw=true)\n\nthen check all the container's logs for being healthy.\n\nConfiguring Microsoft SQL Server to Enable CDC :\n```\n\nCREATE DATABASE Test_DB;\nGO\nUSE Test_DB;\nEXEC sys.sp_cdc_enable_db;ALTER DATABASE Test_DB\nSET CHANGE_TRACKING = ON\n(CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON)\n\n\nCREATE TABLE prospects (\n  id INTEGER IDENTITY(1001,1) NOT NULL PRIMARY KEY,\n  first_name VARCHAR(255) NOT NULL,\n  last_name VARCHAR(255) NOT NULL,\n  email VARCHAR(255) NOT NULL UNIQUE\n);\n\nEXEC sys.sp_cdc_enable_table @source_schema = 'dbo', @source_name = 'prospects', @role_name = NULL, @supports_net_changes = 0;\nGO\n\n```\n\nRestart SQL Server instance and agent \n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/enable-cdc.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/cdc-tables.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/cdc-jobs.png?raw=true)\n\nthen create a kafka connector to sql server :\n\n```\ndocker exec -it connect1 bash\n```\n\n```powershell\n\ncurl -i -X POST -H \"Accept:application/json\" -H \"Content-Type:application/json\" localhost:8083/connectors -d '{ \"name\": \"debezium-connector\", \n\"config\": { \n    \"connector.class\": \"io.debezium.connector.sqlserver.SqlServerConnector\",\n    \"database.hostname\": \"192.168.1.4\", \n    \"database.port\": \"1433\", \n    \"database.user\": \"sa\",\n    \"database.password\": \"ABCabc123456\", \n    \"database.dbname\": \"Test_DB\", \n    \"database.names\":\"Test_DB\",\n    \"database.server.name\": \"192.168.1.4\", \n    \"table.whitelist\": \"dbo.prospects\", \n    \"topic.prefix\": \"fullfillment\",\n    \"database.history.kafka.bootstrap.servers\": \"kafka1:29092\", \n    \"database.history.kafka.topic\": \"schema-changes-topic\",\n    \"errors.log.enable\": \"true\",\n    \"schema.history.internal.kafka.bootstrap.servers\": \"kafka1:29092\",  \n    \"schema.history.internal.kafka.topic\": \"schema-changes.inventory\",\n    \"database.trustServerCertificate\": true  } \n}';\n```\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/connector.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/CreatedConnector.png?raw=true)\n\nand after inserting new data to related sql server table, the connector sync kafka topic with the new data :\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/Insertnew.png?raw=true)\n\ngo to [http://localhost:8080/ui/docker-kafka-server/topic/usertopic/data?sort=Oldest\u0026partition=All](http://localhost:8080/ui/docker-kafka-server/topic/usertopic/data?sort=Oldest\u0026partition=All)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/topicValues.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/topicValues2.png?raw=true)\n\n![alt text](https://raw.githubusercontent.com/kayvansol/kafka-source/main/img/topicValues3.png?raw=true)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayvansol%2Fkafka-source","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkayvansol%2Fkafka-source","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayvansol%2Fkafka-source/lists"}