{"id":21877652,"url":"https://github.com/rbiedrawa/kafka-connect-workshop","last_synced_at":"2026-05-21T05:08:45.380Z","repository":{"id":162193105,"uuid":"379644239","full_name":"rbiedrawa/kafka-connect-workshop","owner":"rbiedrawa","description":"This repository contains the starter code for the `How to play around with Your Apache Kafka Connect cluster?` workshop. This project shows how to install a Kafka Connect plugins and deploy connectors automatically using Docker runtime installation.","archived":false,"fork":false,"pushed_at":"2022-01-31T02:03:44.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-21T23:43:26.727Z","etag":null,"topics":["apache-kafka","connectors","docker","docker-compose","kafka","kafka-connect","kafka-connectors","kafka-ui"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rbiedrawa.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-23T15:11:37.000Z","updated_at":"2021-12-27T12:21:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"d8f8bf39-aea2-4603-b2cb-3292201a6d98","html_url":"https://github.com/rbiedrawa/kafka-connect-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rbiedrawa/kafka-connect-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-connect-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-connect-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-connect-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-connect-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rbiedrawa","download_url":"https://codeload.github.com/rbiedrawa/kafka-connect-workshop/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fkafka-connect-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33289549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T02:57:32.698Z","status":"ssl_error","status_checked_at":"2026-05-21T02:57:31.990Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","connectors","docker","docker-compose","kafka","kafka-connect","kafka-connectors","kafka-ui"],"created_at":"2024-11-28T08:09:52.951Z","updated_at":"2026-05-21T05:08:45.364Z","avatar_url":"https://github.com/rbiedrawa.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Connect Workshop\n\nThis repository contains the starter code for the **How to play around with Your Apache Kafka Connect cluster?** workshop.\n\nThis project shows how to install a Kafka Connect plugins and deploy connectors automatically using Docker runtime\ninstallation.\n\n## Getting Started\n\nUse Docker Compose ([docker-compose.yml](docker-compose.yml)) for setting up and running the whole Apache Kafka Connect\ninfrastructure.\n\nThe following tools are available when you run the whole infrastructure:\n\n* [Kafka Connect](https://kafka.apache.org/documentation/#connect)\n* [Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html)\n* [Kowl UI](https://github.com/cloudhut/kowl)\n* [Kafka Connect UI](https://github.com/lensesio/kafka-connect-ui)\n\n### Prerequisites\n\n* Docker\n\n### Usage\n\n* Run the docker compose stack\n  ```shell\n  docker compose up -d\n  ```\n\n* Check if all components are running\n  ```shell\n  docker compose ps    \n                 \n  # NAME                SERVICE             STATUS              PORTS\n  # kafka               kafka               running             0.0.0.0:9092-\u003e9092/tcp, :::9092-\u003e9092/tcp, 0.0.0.0:9101-\u003e9101/tcp, :::9101-\u003e9101/tcp\n  # kafka-connect       kafka-connect       running (healthy)   0.0.0.0:8083-\u003e8083/tcp, :::8083-\u003e8083/tcp, 9092/tcp\n  # kafka-connect-ui    kafka-connect-ui    running             0.0.0.0:8000-\u003e8000/tcp, :::8000-\u003e8000/tcp\n  # kowl                kowl                running             0.0.0.0:8080-\u003e8080/tcp, :::8080-\u003e8080/tcp\n  # schema-registry     schema-registry     running             0.0.0.0:8081-\u003e8081/tcp, :::8081-\u003e8081/tcp\n  # zookeeper           zookeeper           running             0.0.0.0:2181-\u003e2181/tcp, :::2181-\u003e2181/tcp, 2888/tcp, 3888/tcp  \n  ```\n\n* Use either Curl or Kafka Connect UI to check if autoconfigured `FileStreamSourceConnector` connector is running\n    * Curl\n      ```shell\n      curl -L http://localhost:8083/connectors/fs-test-source-connector | jq\n      \n      # {\n      #   \"name\": \"fs-test-source-connector\",\n      #   \"config\": {\n      #     \"connector.class\": \"org.apache.kafka.connect.file.FileStreamSourceConnector\",\n      #     \"file\": \"/tmp/test.txt\",\n      #     \"tasks.max\": \"1\",\n      #     \"name\": \"fs-test-source-connector\",\n      #     \"topic\": \"test-data\"\n      #   },\n      #   \"tasks\": [\n      #     {\n      #       \"connector\": \"fs-test-source-connector\",\n      #       \"task\": 0\n      #     }\n      #   ],\n      #   \"type\": \"source\"\n      # }\n      ```\n    * [Kafka Connect UI](http://localhost:8000/).\n\n* Verify that custom `Elastic Search Sink` connector is available [here](http://localhost:8000/#/cluster/dev/select-connector) - should be installed during docker image startup is\n\n* Append messages to `/tmp/test.txt` file\n  ```shell\n  docker-compose exec kafka-connect bash -c \"echo 'Hello World 1' \u003e\u003e /tmp/test.txt\"\n  docker-compose exec kafka-connect bash -c \"echo 'Hello World 2' \u003e\u003e /tmp/test.txt\"\n  docker-compose exec kafka-connect bash -c \"echo 'Hello World 3' \u003e\u003e /tmp/test.txt\"\n  ```\n\n* Open your web browser and go to Kowl UI [test-data](http://localhost:8080/topics/test-data) topic\n\n* Stop docker compose stack\n  ```shell\n  docker compose down -v\n  ```\n\n## FAQ\n\n### How to install connectors differently than using Docker runtime ??\n\n1) Custom Kafka Connect image\n\n```shell\nFROM confluentinc/cp-schema-registry:6.2.0\nENV CONNECT_PLUGIN_PATH=\"/usr/share/java,/usr/share/confluent-hub-components\"\n\nRUN confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:11.0.6\n```\n\n2) Docker volume mapping\n\n```shell\n    …\n    environment:\n      …\n      CONNECT_PLUGIN_PATH: '/usr/share/java,/data/connectors/'\n    volumes:\n      - /path/to/custom/connectors/folder:/data/connectors/\n```\n\n## Important Endpoints\n\n| Name | Endpoint | \n| -------------:|:--------:|\n| `Kafka Connect` | [http://localhost:8083/](http://localhost:8083/) |\n| `Kafka Connect UI` | [http://localhost:8000/](http://localhost:8000/) |\n| `Kowl UI` | [http://localhost:8080/](http://localhost:8080/) |\n| `Schema-registry` | [http://localhost:8081/](http://localhost:8081/) |\n\n## References\n\n* [Kafka Connect](https://github.com/confluentinc/kafka-images/tree/master/kafka-connect)\n* [Confluent Kafka Connect](https://docs.confluent.io/current/connect/index.html)\n* [FileSystem Source Connector](https://www.confluent.io/hub/mmolimar/kafka-connect-fs)\n* [Kafka Docker Images](https://github.com/confluentinc/kafka-images)\n* [Kafka Connect UI](https://github.com/lensesio/kafka-connect-ui)\n* [cloudhut/kowl](https://github.com/cloudhut/kowl)\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fkafka-connect-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frbiedrawa%2Fkafka-connect-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fkafka-connect-workshop/lists"}