{"id":22152128,"url":"https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink","last_synced_at":"2025-07-26T05:32:04.115Z","repository":{"id":37985298,"uuid":"258697838","full_name":"findinpath/kafka-connect-nested-set-jdbc-sink","owner":"findinpath","description":"JDBC Nested Set Sink Connector for Kafka Connect","archived":false,"fork":false,"pushed_at":"2023-02-27T03:56:53.000Z","size":1462,"stargazers_count":4,"open_issues_count":8,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-06-30T07:05:04.846Z","etag":null,"topics":["jdbc","kafka","kafka-connector","nested-set","streaming"],"latest_commit_sha":null,"homepage":"https://www.findinpath.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-25T05:49:34.000Z","updated_at":"2021-12-07T01:09:26.000Z","dependencies_parsed_at":"2023-01-27T12:16:24.438Z","dependency_job_id":null,"html_url":"https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fkafka-connect-nested-set-jdbc-sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fkafka-connect-nested-set-jdbc-sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fkafka-connect-nested-set-jdbc-sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fkafka-connect-nested-set-jdbc-sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/kafka-connect-nested-set-jdbc-sink/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227652564,"owners_count":17799235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jdbc","kafka","kafka-connector","nested-set","streaming"],"created_at":"2024-12-02T00:47:18.246Z","updated_at":"2024-12-02T00:47:19.106Z","avatar_url":"https://github.com/findinpath.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](http://img.shields.io/:license-apache%202.0-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\n[![Build Status](https://travis-ci.com/findinpath/kafka-connect-nested-set-jdbc-sink.svg?branch=master)](https://travis-ci.com/findinpath/kafka-connect-nested-set-jdbc-sink)\n\n\nLicensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).\n\nJDBC Nested Set Sink Connector for Kafka Connect\n================================================\n\nThis is a [Kafka Connector](http://kafka.apache.org/documentation.html#connect)\nfor loading [nested set model](https://en.wikipedia.org/wiki/Nested_set_model) to any JDBC compatible database.\n\n\nThe Confluent [kafka-connect-jdbc sink connector](https://docs.confluent.io/current/connect/kafka-connect-jdbc/sink-connector/index.html) \nwas already developed to allow exporting of data from Apache Kafka topics to any relational database.\n\nThis connector builts on top of the existing `kakfa-connect-jdbc sink connector` \nfunctionality and  provides a safeguard on the sink \ndatabase side in order to avoid adding invalid updates to the nested set model.\nThe sinked nested set records will be placed in an `INSERT only` log table\nand only when the nested set configuration is valid, the log records get\ninserted to the destination sink table.\n\n\nThe project source code is a port of the Confluent's \n[kafka-connect-jdbc](https://github.com/confluentinc/kafka-connect-jdbc)\nadapted to consistently sink nested set data.\nThis is why large portions of the `kafka-connect-jdbc` project code have\nbeen copied and adapted.\n\n**NOTE** : This connector has been extensively tested against Confluent 5.5.0\n\n## Nested set model\n\nAs stated on [Wikipedia](https://en.wikipedia.org/wiki/Nested_set_model)\n\u003e The nested set model is a technique for representing nested sets (also known as trees or hierarchies) in relational databases.\n\nDetails about the advantages of the nested set model are very well \ndescribed in the following article:\n\nhttps://www.sitepoint.com/hierarchical-data-database/\n\n![Nested Set Tree](https://raw.githubusercontent.com/findinpath/kafka-connect-nested-set-jdbc-sink/master/images/nested-set.gif)\n\n![Nested Set Model](https://raw.githubusercontent.com/findinpath/kafka-connect-nested-set-jdbc-sink/master/images/nested-set-model.png)\n\n\n##Syncing nested set models over Apache Kafka\n\nBellow is described a concrete example to showcase the complexity associated\nwith syncing a nested set model over Apache Kafka.\n\nLet's take the following example of going from a nested set model \nconsisting only of the root node:\n\n```\n|1| A |2|\n```\n\nto the nested set model (after adding two children) consisting of 3 nodes:\n\n```\n|1| A |6|\n    ├── |2| B |3|\n    └── |4| C |5|\n```\n\nIn the snippets above the tree node labels are prefixed by their `left` and `right`\nnested set model coordinates.\n\nVia `kafka-connect-jdbc` the records corresponding to the tuple updates may come in various\nordering:\n\n```\n| label | left | right |\n|-------|------|-------|\n| A     | 1    | 6     |\n| B     | 2    | 3     |\n| C     | 4    | 5     |\n```\nor\n```\n| label | left | right |\n|-------|------|-------|\n| B     | 2    | 3     |\n| C     | 4    | 5     |\n| A     | 1    | 6     |\n```\nor, in case that there's a problem on the jdbc source connector's side\n```\n| label | left | right |\n|-------|------|-------|\n| B     | 2    | 3     |\n| C     | 4    | 5     |\n```\n\nor any other combinations of the three tuples listed above because the records\nare polled in batches of different sizes from Apache Kafka. \n\n\nGoing from the nested set model table content\n\n```\n| label | left | right |\n|-------|------|-------|\n| A     | 1    | 2     |\n```\ntowards\n\n```\n| label | left | right |\n|-------|------|-------|\n| A     | 1    | 6     |\n```\n\nor \n\n```\n| label | left | right |\n|-------|------|-------|\n| A     | 1    | 6     |\n| B     | 2    | 3     |\n```\n\nwould intermittently render the nested set model corrupt until\nall the records from the source nested set model are synced over Apache Kafka.\nIn the unlikely case that there's an outage during a tree update on the Kafka Connect JDBC \nsource connector side, the nested set model from the sink database side would stay corrupt\nuntil the outage is fixed or until the nested set model would be manually adjusted. \n\n\nOne solution to cope with this sync problem would be to separate \nthe nested set model from what is synced over Apache Kafka. \n\n![Sink database table diagram](https://raw.githubusercontent.com/findinpath/kafka-connect-nested-set-jdbc-sink/master/images/sink-database-table-diagram.png)\n\n\nIn the table diagram above, the `nested_set_node_log` table is an `INSERT only` table\nin which is written whenever a new record(s) is read from Apache Kafka.\nThe `log_offset` table has only one tuple pointing to the last `nested_set_node_log` tuple id\nprocessed when updating the `nested_set_node` table.\n\nWhenever new records are read from Apache Kafka, there will be a transactional attempt \nto apply all the updates from `nested_set_node_log` made after the saved entry in the `log_offset`\ntable to the existing configuration of the `nested_set_node` nested set model.\n\nIf the applied updates lead to a valid nested set model configuration, then the `nested_set_node`\ntable will be updated and the log offset will be set to the latest processed `nested_set_node_log` entry.\nOtherwise the `nested_set_node` table stays in its previous state. \n\n\n## Installation notes\n\nThis component is available on the [Confluent Hub](https://www.confluent.io/hub/findinpath/kafka-connect-nested-set-jdbc-sink).\n\nTherefor it can be easily installed with the following command: \n\n```\nconfluent-hub install findinpath/kafka-connect-nested-set-jdbc-sink:1.0.0\n```\n\n**NOTE** : This connector has been extensively tested against Confluent 5.5.0 on JDK 8.\n\n\n## Connector configuration\n\nThe connector configuration can be obtained by doing the following call on Kafka Connect REST API:\n\n```bash\ncurl -s -X PUT http://localhost:8083/connector-plugins/NestedSetJdbcSinkConnector/config/validate --header 'Content-Type: application/json' -d $'{\"connector.class\": \"com.findinpath.connect.nestedset.jdbc.NestedSetJdbcSinkConnector\", \"topics\": \"dummy\"}' | jq\n```\n\n\nA sample configuration for the connector is shown below:\n\n```json\n{\n    \"name\": \"findinpath-sink\",\n    \"config\": {\n        \"name\": \"findinpath-sink\",\n        \"connector.class\": \"com.findinpath.connect.nestedset.jdbc.NestedSetJdbcSinkConnector\",\n        \"tasks.max\": \"1\",\n        \"topics\": \"findinpath.nested_set_node\",\n        \"connection.url\": \"jdbc:postgresql://sink:5432/sink?loggerLevel=OFF\",\n        \"connection.user\": \"sa\",\n        \"connection.password\": \"p@ssw0rd!sink\",\n        \"pk.fields\": \"id\",\n        \"table.name\": \"nested_set_node\",\n        \"table.left.column.name\": \"lft\",\n        \"table.rgt.column.name\": \"rgt\",\n        \"log.table.name\": \"nested_set_node_log\",\n        \"log.table.primary.key.column.name\": \"log_id\",\n        \"log.table.operation.type.column.name\": \"operation_type\",\n        \"log.offset.table.name\": \"nested_set_node_log_offset\",\n        \"log.offset.table.log.table.column.name\": \"log_table_name\",\n        \"log.offset.table.offset.column.name\": \"log_table_offset\"\n    }\n}\n```\n\n## Quick start\n\nCheckout the following resources:\n\n- [TESTING_UPSERT.md](https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink/blob/master/TESTING_UPSERT.md) for getting a grip on how to use the connector for syncing *upserts* (`INSERT`/`UPDATE` sql statements) in the nested set model\n- [TESTING_DELETE.md](https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink/blob/master/TESTING_DELETE.md) for getting a grip on how to use the connector for syncing *deletes* (`DELETE` sql statements) in the nested set model\n\n\n## End to end tests\n\nThis project contains end to end tests for the _kafka-connect-nested-set-jdbc-sink_ connector:\n\n![kafka-connect-nested-set-jdbc-sink End to End Test Architecture](https://raw.githubusercontent.com/findinpath/kafka-connect-nested-set-jdbc-sink/master/images/end-to-end-test-architecture-diagram.png)\n\nCheck [DemoNestedSetSyncTest.java](https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink/blob/master/src/test/java/com/findinpath/connect/nestedset/jdbc/DemoNestedSetSyncTest.java)\nfor implementation details.\n\n## Contribute\n\nSource code: https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink\n\nIssue Tracker: https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink/issues\n\n## License\n\nThis project is licensed under the [Apache License](https://github.com/findinpath/kafka-connect-nested-set-jdbc-sink/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fkafka-connect-nested-set-jdbc-sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Fkafka-connect-nested-set-jdbc-sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fkafka-connect-nested-set-jdbc-sink/lists"}