{"id":18048075,"url":"https://github.com/ivangfr/springboot-kafka-connect-jdbc-streams","last_synced_at":"2025-04-05T09:04:54.045Z","repository":{"id":47402843,"uuid":"165393210","full_name":"ivangfr/springboot-kafka-connect-jdbc-streams","owner":"ivangfr","description":"Project goal: Explore Kafka, Kafka Connect, and Kafka Streams. Components: store-api: Inserts/updates MySQL records. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors: Listen to Kafka, insert/update Elasticsearch docs. store-streams: Listens to Kafka, processes with Kafka Streams, pushes new messages to Kafka.","archived":false,"fork":false,"pushed_at":"2025-03-02T15:29:49.000Z","size":297259,"stargazers_count":113,"open_issues_count":0,"forks_count":72,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-29T08:03:02.293Z","etag":null,"topics":["avro","docker","elasticsearch","java","jib","json","kafka","kafka-connect","kafka-streams","mysql","schema-registry","spring-boot","spring-cloud-stream","spring-data-jpa","spring-web-mvc","springdoc-openapi"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivangfr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"ivangfr"}},"created_at":"2019-01-12T13:51:21.000Z","updated_at":"2025-03-05T09:18:44.000Z","dependencies_parsed_at":"2024-04-27T22:36:22.485Z","dependency_job_id":null,"html_url":"https://github.com/ivangfr/springboot-kafka-connect-jdbc-streams","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspringboot-kafka-connect-jdbc-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspringboot-kafka-connect-jdbc-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspringboot-kafka-connect-jdbc-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspringboot-kafka-connect-jdbc-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivangfr","download_url":"https://codeload.github.com/ivangfr/springboot-kafka-connect-jdbc-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312068,"owners_count":20918344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","docker","elasticsearch","java","jib","json","kafka","kafka-connect","kafka-streams","mysql","schema-registry","spring-boot","spring-cloud-stream","spring-data-jpa","spring-web-mvc","springdoc-openapi"],"created_at":"2024-10-30T20:11:19.555Z","updated_at":"2025-04-05T09:04:53.987Z","avatar_url":"https://github.com/ivangfr.png","language":"Java","funding_links":["https://github.com/sponsors/ivangfr"],"categories":[],"sub_categories":[],"readme":"# springboot-kafka-connect-jdbc-streams\n\nThe main goal of this project is to explore [`Kafka`](https://kafka.apache.org), [`Kafka Connect`](https://docs.confluent.io/platform/current/connect/index.html), and [`Kafka Streams`](https://docs.confluent.io/platform/current/streams/overview.html). The project includes: `store-api`, which inserts/updates records in [`MySQL`](https://www.mysql.com); `Source Connectors` that monitor these records in `MySQL` and push related messages to `Kafka`; `Sink Connectors` that listen to messages from `Kafka` and insert/update documents in [`Elasticsearch`](https://www.elastic.co); and `store-streams`, which listens to messages from `Kafka`, processes them using `Kafka Streams`, and pushes new messages back to `Kafka`.\n\n## Proof-of-Concepts \u0026 Articles\n\nOn [ivangfr.github.io](https://ivangfr.github.io), I have compiled my Proof-of-Concepts (PoCs) and articles. You can easily search for the technology you are interested in by using the filter. Who knows, perhaps I have already implemented a PoC or written an article about what you are looking for.\n\n## Additional Readings\n\n- \\[**Medium**\\] [**Streaming MySQL changes to Elasticsearch using Kafka Connect**](https://medium.com/@ivangfr/streaming-mysql-changes-to-elasticsearch-using-kafka-connect-fe22a5d0aa27)\n- \\[**Medium**\\] [**Enhancing a MySQL-KafkaConnect-Elasticsearch Setup with Spring Boot Applications**](https://medium.com/@ivangfr/enhancing-a-mysql-kafkaconnect-elasticsearch-setup-with-spring-boot-applications-257c65ff0965)\n\n## Project Diagram\n\n![project-diagram](documentation/project-diagram.png)\n\n## Applications\n\n- ### store-api\n\n  Monolithic [`Spring Boot`](https://docs.spring.io/spring-boot/index.html) application that exposes a REST API to manage `Customers`, `Products` and `Orders`. The data is stored in `MySQL`.\n  \n  ![store-api-swagger](documentation/store-api-swagger.jpeg)\n\n- ### store-streams\n\n  `Spring Boot` application that connects to `Kafka` and uses `Kafka Streams API` to transform some _\"input\"_ topics into a new _\"output\"_ topic in `Kafka`.\n\n## Prerequisites\n\n- [`Java 21`](https://www.oracle.com/java/technologies/downloads/#java21) or higher;\n- A containerization tool (e.g., [`Docker`](https://www.docker.com), [`Podman`](https://podman.io), etc.)\n\n## (De)Serialization formats\n\nIn order to run this project, you can use [`JSON`](https://www.json.org) or [`Avro`](https://avro.apache.org) format to serialize/deserialize data to/from the `binary` format used by Kafka. The default format is `JSON`. Throughout this document, I will point out what to do if you want to use `Avro`.\n\n## Start Environment\n\n- Open a terminal and inside the `springboot-kafka-connect-jdbc-streams` root folder run:\n  ```\n  docker compose up -d\n  ```\n  \u003e **Note**: During the first run, an image for `kafka-connect` will be built with the name `springboot-kafka-connect-jdbc-streams_kafka-connect`. Use the command below to rebuild it.\n  \u003e ```\n  \u003e docker compose build\n  \u003e ```\n\n- Wait for all Docker containers to be up and running. To check it, run:\n  ```\n  docker ps -a\n  ```\n  \n## Create Kafka Topics\n\nIn order to have topics in `Kafka` with more than `1` partition, we have to create them manually and not let the connectors to create them for us. So, for it:\n\n- Open a new terminal and make sure you are in the `springboot-kafka-connect-jdbc-streams` root folder;\n\n- Run the script below:\n  ```\n  ./create-kafka-topics.sh\n  ```\n\n  It will create the topics `mysql.storedb.customers`, `mysql.storedb.products`, `mysql.storedb.orders`, `mysql.storedb.orders_products` with `5` partitions.\n\n## Create connectors\n\nConnectors use `Converters` for data serialization and deserialization. If you are configuring `For JSON (de)serialization`, the converter used is `JsonConverter`. On the other hand, the converter used is `AvroConverter`.\n\n\u003e **Important**: If the `Source Connector Converter` serializes data (e.g., from `JSON` to `bytes` using `JsonConverter`), the `Sink Connector Converter` must also use `JsonConverter` to deserialize the `bytes`. Otherwise, an error will be thrown. The document [Kafka Connect Deep Dive – Converters and Serialization Explained](https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained) explains this in detail.\n\nSteps to create the connectors:\n\n- In a terminal, navigate to the `springboot-kafka-connect-jdbc-streams` root folder\n\n- Run the following script to create the connectors on `kafka-connect`:\n\n  - **For JSON (de)serialization**\n\n    ```\n    ./create-connectors-jsonconverter.sh\n    ```\n    \n  - **For Avro (de)serialization**\n\n    ```\n    ./create-connectors-avroconverter.sh\n    ```\n\n- You can check the state of the connectors and their tasks on `Kafka Connect UI` or running the following script:\n  ```\n  ./check-connectors-state.sh\n  ```\n\n- Once the connectors and their tasks are ready (`RUNNING` state), you should see something like:\n  ```\n  {\"name\":\"mysql-source-customers\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"source\"}\n  {\"name\":\"mysql-source-products\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"source\"}\n  {\"name\":\"mysql-source-orders\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"source\"}\n  {\"name\":\"mysql-source-orders_products\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"source\"}\n  {\"name\":\"elasticsearch-sink-customers\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"sink\"}\n  {\"name\":\"elasticsearch-sink-products\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"sink\"}\n  {\"name\":\"elasticsearch-sink-orders\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"kafka-connect:8083\"}],\"type\":\"sink\"}\n  ```\n\n- On `Kafka Connect UI` (http://localhost:8086), you should see:\n\n  ![kafka-connect-ui](documentation/kafka-connect-ui.jpeg)\n\n- If there is any problem, you can check `kafka-connect` container logs:\n  ```\n  docker logs kafka-connect\n  ```\n## Running Applications with Maven\n\n- **store-api**\n\n  - Open a new terminal and make sure you are in the `springboot-kafka-connect-jdbc-streams` root folder.\n  \n  - Run the command below to start the application:\n    ```\n    ./mvnw clean spring-boot:run --projects store-api \\\n    -Dspring-boot.run.jvmArguments=\"-Dserver.port=9080\"\n    ```\n    \u003e **Note**\n    \u003e\n    \u003e It will create all tables, such as: `customers`, `products`, `orders` and `orders_products`. We are using `spring.jpa.hibernate.ddl-auto=update` configuration.\n    \u003e \n    \u003e It will also insert some customers and products. If you don't want it, just set to `false` the properties `load-samples.customers.enabled` and `load-samples.products.enabled` in `application.yml`.\n\n- **store-streams**\n\n  - Open a new terminal and inside the `springboot-kafka-connect-jdbc-streams` root folder.\n\n  - To start application, run:\n\n    - **For JSON (de)serialization**\n  \n      ```\n      ./mvnw clean spring-boot:run --projects store-streams \\\n      -Dspring-boot.run.jvmArguments=\"-Dserver.port=9081\"\n      ```\n      \n    - **For Avro (de)serialization**\n     \n      \u003e **Warning**: Unable to run in this mode on my machine! The application starts fine when using the `avro` profile, but when the first event arrives, the `org.apache.kafka.common.errors.SerializationException: Unknown magic byte!` is thrown. This problem does not occur when [Running Applications as Docker containers](#running-applications-as-docker-containers).\n      ```\n      ./mvnw clean spring-boot:run --projects store-streams \\\n      -Dspring-boot.run.jvmArguments=\"-Dserver.port=9081\" \\\n      -Dspring-boot.run.profiles=avro\n      ```\n      \u003e The command below generates Java classes from Avro files present in `src/main/resources/avro`\n      \u003e ```\n      \u003e ./mvnw generate-sources --projects store-streams\n      \u003e ```\n\n## Running Applications as Docker containers\n\n### Build Application’s Docker Image\n\n- In a terminal, make sure you are inside the `springboot-kafka-connect-jdbc-streams` root folder;\n\n- Run the following script to build the application's docker image:\n  ```\n  ./build-docker-images.sh\n  ```\n\n### Application’s Environment Variables\n\n- **store-api**\n\n  | Environment Variable   | Description                                                       |\n  |------------------------|-------------------------------------------------------------------|\n  | `MYSQL_HOST`           | Specify host of the `MySQL` database to use (default `localhost`) |\n  | `MYSQL_PORT`           | Specify port of the `MySQL` database to use (default `3306`)      |\n\n- **store-streams**\n\n  | Environment Variable   | Description                                                             |\n  |------------------------|-------------------------------------------------------------------------|\n  | `KAFKA_HOST`           | Specify host of the `Kafka` message broker to use (default `localhost`) |\n  | `KAFKA_PORT`           | Specify port of the `Kafka` message broker to use (default `29092`)     |\n  | `SCHEMA_REGISTRY_HOST` | Specify host of the `Schema Registry` to use (default `localhost`)      |\n  | `SCHEMA_REGISTRY_PORT` | Specify port of the `Schema Registry` to use (default `8081`)           |\n\n### Run Application’s Docker Container\n\n- In a terminal, make sure you are inside the `springboot-kafka-connect-jdbc-streams` root folder;\n\n- In order to run the application's docker containers, you can pick between `JSON` or `Avro`:\n\n  - **For JSON (de)serialization**\n    ```\n    ./start-apps.sh\n    ```\n  - **For Avro (de)serialization**\n    ```\n    ./start-apps.sh avro\n    ```\n\n## Application's URL\n\n| Application   | URL                                   |\n|---------------|---------------------------------------|\n| store-api     | http://localhost:9080/swagger-ui.html |\n| store-streams | http://localhost:9081/actuator/health |\n\n## Testing\n\n1. Let's simulate an order creation. In this example, customer with id `1`\n   ```\n   {\"id\":1, \"name\":\"John Gates\", \"email\":\"john.gates@test.com\", \"address\":\"street 1\", \"phone\":\"112233\"}\n   ```\n   will order one unit of the product with id `15`\n   ```\n   {\"id\":15, \"name\":\"iPhone Xr\", \"price\":900.00}\n   ```\n\n   In a terminal, run the following `curl` command:\n   ```\n   curl -i -X POST localhost:9080/api/orders \\\n     -H 'Content-Type: application/json' \\\n     -d '{\"customerId\": 1, \"paymentType\": \"BITCOIN\", \"status\": \"OPEN\", \"products\": [{\"id\": 15, \"unit\": 1}]}'\n   ```\n\n   The response should be:\n   ```\n   HTTP/1.1 201\n   {\n     \"id\": \"47675629-4f0d-440d-b6df-c829874ee2a6\",\n     \"customerId\": 1,\n     \"paymentType\": \"BITCOIN\",\n     \"status\": \"OPEN\",\n     \"products\": [{\"id\": 15, \"unit\": 1}]\n   }\n   ```\n\n2. Checking `Elasticsearch`:\n   ```\n   curl \"localhost:9200/store.streams.orders/_search?pretty\"\n   ```\n   \n   We should have one order with a customer and products names:\n   ```\n   {\n     \"took\" : 844,\n     \"timed_out\" : false,\n     \"_shards\" : {\n       \"total\" : 1,\n       \"successful\" : 1,\n       \"skipped\" : 0,\n       \"failed\" : 0\n     },\n     \"hits\" : {\n       \"total\" : {\n         \"value\" : 1,\n         \"relation\" : \"eq\"\n       },\n       \"max_score\" : 1.0,\n       \"hits\" : [\n         {\n           \"_index\" : \"store.streams.orders\",\n           \"_type\" : \"order\",\n           \"_id\" : \"47675629-4f0d-440d-b6df-c829874ee2a6\",\n           \"_score\" : 1.0,\n           \"_source\" : {\n             \"payment_type\" : \"BITCOIN\",\n             \"created_at\" : 1606821792360,\n             \"id\" : \"47675629-4f0d-440d-b6df-c829874ee2a6\",\n             \"customer_name\" : \"John Gates\",\n             \"customer_id\" : 1,\n             \"status\" : \"OPEN\",\n             \"products\" : [\n               {\n                 \"unit\" : 1,\n                 \"price\" : 900,\n                 \"name\" : \"iPhone Xr\",\n                 \"id\" : 15\n               }\n             ]\n           }\n         }\n       ]\n     }\n   }\n   ```\n\n3. In order to create random orders, we can use also the `simulation`:\n   ```\n   curl -i -X POST localhost:9080/api/simulation/orders \\\n     -H 'Content-Type: application/json' \\\n     -d '{\"total\": 10, \"sleep\": 100}'\n   ```\n\n## Useful Links/Commands\n\n- **Kafka Topics UI**\n\n  `Kafka Topics UI` can be accessed at http://localhost:8085\n\n- **Kafka Connect UI**\n\n  `Kafka Connect UI` can be accessed at http://localhost:8086\n\n- **Schema Registry UI**\n\n  `Schema Registry UI` can be accessed at http://localhost:8001\n\n- **Schema Registry**\n\n  You can use `curl` to check the subjects in `Schema Registry`\n\n  - Get the list of subjects\n    ```\n    curl localhost:8081/subjects\n    ```\n  - Get the latest version of the subject `mysql.storedb.customers-value`\n    ```\n    curl localhost:8081/subjects/mysql.storedb.customers-value/versions/latest\n    ```\n\n- **Kafka Manager**\n\n  `Kafka Manager` can be accessed at http://localhost:9000\n\n  _Configuration_\n  - First, you must create a new cluster. Click on `Cluster` (dropdown on the header) and then on `Add Cluster`;\n  - Type the name of your cluster in `Cluster Name` field, for example: `MyCluster`;\n  - Type `zookeeper:2181` in `Cluster Zookeeper Hosts` field;\n  - Enable checkbox `Poll consumer information (Not recommended for large # of consumers if ZK is used for offsets tracking on older Kafka versions)`;\n  - Click on `Save` button at the bottom of the page.\n\n- **Elasticsearch**\n\n  `Elasticsearch` can be accessed at http://localhost:9200\n\n  - Get all indices:\n    ```\n    curl \"localhost:9200/_cat/indices?v\"\n    ```\n  - Search for documents:\n    ```\n    curl \"localhost:9200/mysql.storedb.customers/_search?pretty\"\n    curl \"localhost:9200/mysql.storedb.products/_search?pretty\"\n    curl \"localhost:9200/store.streams.orders/_search?pretty\"\n    ```\n\n- **MySQL**\n\n  ```\n  docker exec -it -e MYSQL_PWD=secret mysql mysql -uroot --database storedb\n  select * from orders;\n  ```\n  \n## Shutdown\n\n- To stop applications:\n  - If they were started with `Maven`, go to the terminals where they are running and press `Ctrl+C`;\n  - If they were started as Docker containers, go to a terminal and, inside the `springboot-kafka-connect-jdbc-streams` root folder, run the script below:\n    ```\n    ./stop-apps.sh\n    ```\n- To stop and remove docker compose containers, network and volumes, go to a terminal and, inside the `springboot-kafka-connect-jdbc-streams` root folder, run the following command:\n  ```\n  docker compose down -v\n  ```\n\n## Cleanup\n\nTo remove the Docker images created by this project, go to a terminal and, inside the `springboot-kafka-connect-jdbc-streams` root folder, run the script below:\n```\n./remove-docker-images.sh\n```\n\n## Issues\n\n- Product `price` field, [numeric.mapping doesn't work for DECIMAL fields #563](https://github.com/confluentinc/kafka-connect-jdbc/issues/563). For now, the workaround is using `String` instead of `BigDecimal` as type for this field.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangfr%2Fspringboot-kafka-connect-jdbc-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivangfr%2Fspringboot-kafka-connect-jdbc-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangfr%2Fspringboot-kafka-connect-jdbc-streams/lists"}