{"id":22152135,"url":"https://github.com/findinpath/testcontainers-kafka-connect","last_synced_at":"2025-07-26T05:32:09.294Z","repository":{"id":43851101,"uuid":"254327398","full_name":"findinpath/testcontainers-kafka-connect","owner":"findinpath","description":"kafka-connect-jdbc system test based on testcontainers ","archived":false,"fork":false,"pushed_at":"2022-09-08T01:07:31.000Z","size":1009,"stargazers_count":9,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-03-05T15:35:32.533Z","etag":null,"topics":["kafka","kafka-connect","kafka-connect-jdbc","testcontainers"],"latest_commit_sha":null,"homepage":"https://www.findinpath.com/kafka-connect-jdbc-system-tests/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-09T09:27:13.000Z","updated_at":"2023-01-27T11:19:27.000Z","dependencies_parsed_at":"2023-01-18T00:00:48.917Z","dependency_job_id":null,"html_url":"https://github.com/findinpath/testcontainers-kafka-connect","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftestcontainers-kafka-connect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftestcontainers-kafka-connect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftestcontainers-kafka-connect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Ftestcontainers-kafka-connect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/testcontainers-kafka-connect/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227652564,"owners_count":17799235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-connect","kafka-connect-jdbc","testcontainers"],"created_at":"2024-12-02T00:47:20.879Z","updated_at":"2024-12-02T00:47:28.906Z","avatar_url":"https://github.com/findinpath.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Consumption of Apache Kafka AVRO entities synced via kafka-connect-jdbc from PostgreSQL \n=======================================================================================\n\nThis is a showcase on how to test the synchronization of the contents\nof a Postgres table via [kafka-connect-jdbc](https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html)\ntowards [Apache Kafka](https://kafka.apache.org/).\n\nThe contents of the input Postgres table are synced in  [AVRO](https://avro.apache.org/) messages \ntowards Apache Kafka.\n \nThe current project makes use of [docker](https://www.docker.com/) containers \n(via [testcontainers](https://www.testcontainers.org/) library) for showcasing\nthe Confluent Kakfa Connect functionality in an automated test case.\n\nThe interactions from this proof of concept are described visually in the image below:\n\n![kafka-connect-jdbc System Test Architecture](images/kafka-connect-jdbc_system-test_architecture.png)\n\nThis proof of concept can be used for making end to end system test cases with\n[docker](https://www.docker.com/) for architectures that rely on \n[kafka-connect-jdbc](https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html).\n\n \nThe [testcontainers](https://www.testcontainers.org/) library already\noffers a [Kafka](https://www.testcontainers.org/modules/kafka/) module\nfor interacting with [Apache Kafka](https://kafka.apache.org/), but\nthere is not, at the moment, a testcontainers module for the whole\nConfluent environment (Confluent Schema Registry container support is\nmissing from the module previously mentioned).\nAs a side note, the containers used do not use the default ports exposed\nby default in the artifacts (e.g. : Apache Zookeeper 2181, Apache Kafka 9092,\nConfluent Schema Registry 8081, Apache Kafka Connect 8083), but rather free ports available on the\ntest machine avoiding therefor possible conflicts with already running\nservices on the test machine. \n\n\n**NOTE** In order to interact with PostgreSQL from **Apache Kafka Connect** the PostgreSQL\ndriver (see [postgresql-42.2.12.jar](src/test/resources/plugins/kafka-connect-jdbc/postgresql-42.2.12.jar))\nis installed in the **Apache Kafka Connect* testcontainer.\nSee the [KafkaConnectDemoTest.java](src/test/java/com/findinpath/KafkaConnectDemoTest.java) for details on how\nto [KafkaConnectContainer](src/test/java/com/findinpath/testcontainers/KafkaConnectContainer.java) is initialized\nwith this plugin.\n\nMore details on installing Apache Kafka Connect plugins are available\n [here](https://docs.confluent.io/current/connect/userguide.html#connect-installing-plugins).\n \n \nFor the test environment the following containers will be started:\n \n- Apache Zookeeper\n- Apache Kafka\n- Confluent Schema Registry\n- Confluent Kafka Connect\n- PostgreSQL\n\nIt is quite impressive to see how close a productive environment can be simulated in the test cases  \nwith the [testcontainers](https://www.testcontainers.org/) library.\n \nOnce the test environment is started, a kafka-connect-jdbc connector will be registered \nfor the `bookmarks` PostgreSQL table.\nThe kafka-connect-jdbc connector will then poll the `bookmarks` table and will sync\nits contents towards the `findinpath.bookmarks` Apache Kafka topic.\n\nThe test demo verifies whether the dynamically inserted contents \ninto the `bookmarks` Postgres table get successfully synced in \nAVRO format on the Apache Kafka topic `findinpath.bookmarks` in the \nsame order as they were inserted. \n\nUse \n\n```bash\nmvn clean install\n```\n\nfor trying out the project.\n\nThe file [testcontainers.properties](src/test/resources/testcontainers.properties) can be\nused for overriding the default [docker](https://www.docker.com/) images used for the containers needed in setting \nup the Confluent test environment.\n\n\n## Test containers\n\nThis project provides a functional prototype on how to setup the whole\nConfluent environment (including **Confluent Schema Registry** and **Apache Kafka Connect**) \nvia testcontainers.\n\n### Kafka Connect\n\nIn order to use the Confluent's Kafka Connect container, this project made use of the already existing code\nfor [KafkaConnectContainer](https://github.com/ydespreaux/testcontainers/blob/master/testcontainers-kafka/src/main/java/com/github/ydespreaux/testcontainers/kafka/containers/KafkaConnectContainer.java)\non [ydespreaux](https://github.com/ydespreaux) Github account.\n\n**NOTE** that the `KafkaConnectContainer` class previously mentioned has also corresponding test cases \nwithin the project [lib-kafka-connect](https://github.com/ydespreaux/shared/tree/master/lib-kafka-connect) in order to have a clue\non how to interact in an integration test with the container.  ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Ftestcontainers-kafka-connect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Ftestcontainers-kafka-connect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Ftestcontainers-kafka-connect/lists"}