{"id":26164898,"url":"https://github.com/angular2guy/simpledatastreampipelines","last_synced_at":"2025-10-12T20:09:04.810Z","repository":{"id":270200634,"uuid":"909605653","full_name":"Angular2Guy/SimpleDataStreamPipelines","owner":"Angular2Guy","description":"Simple Efficient Spring/Kafka DataStreams. Datastreams have a source to create Kafka events from and send them to Kafka. The DataStream sink receives the events and sends them to the destination. ","archived":false,"fork":false,"pushed_at":"2025-10-07T17:24:46.000Z","size":162,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-07T18:39:58.279Z","etag":null,"topics":["actuator-spring-boot","debezium","events","helm-chart","java","jpa","kafka","kotlin","kubernetes","minikube","postgresql","rest","soap-web-services","spring-boot"],"latest_commit_sha":null,"homepage":"https://angular2guy.wordpress.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Angular2Guy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-29T08:18:46.000Z","updated_at":"2025-10-07T17:24:49.000Z","dependencies_parsed_at":"2024-12-29T10:24:52.897Z","dependency_job_id":"9d202022-218a-4e1b-a493-bbd1c7e1a8fd","html_url":"https://github.com/Angular2Guy/SimpleDataStreamPipelines","commit_stats":null,"previous_names":["angular2guy/simpledatastreampipelines"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Angular2Guy/SimpleDataStreamPipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angular2Guy%2FSimpleDataStreamPipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angular2Guy%2FSimpleDataStreamPipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angular2Guy%2FSimpleDataStreamPipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angular2Guy%2FSimpleDataStreamPipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Angular2Guy","download_url":"https://codeload.github.com/Angular2Guy/SimpleDataStreamPipelines/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angular2Guy%2FSimpleDataStreamPipelines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279012794,"owners_count":26085187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-12T02:00:06.719Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actuator-spring-boot","debezium","events","helm-chart","java","jpa","kafka","kotlin","kubernetes","minikube","postgresql","rest","soap-web-services","spring-boot"],"created_at":"2025-03-11T15:57:12.971Z","updated_at":"2025-10-12T20:09:04.805Z","avatar_url":"https://github.com/Angular2Guy.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SimpeDataStreamPipelines\n\n---\n\nThis is a project shows howto create DataStream Pipelines based on Spring Boot and Kafka \nthat are simple, efficient and based on Jdk 24+. A DataStream Pipeline has a source that\nreceives changes/requests and transforms them to events that are send to Kafka. The \nKafka events are received by the sink and transformed to be send to the receiving\nsystem. A DataStream is one Spring Boot application providing the source and sink.\nThe scalability and efficiency is provided by Jdk 24+ Virtual Threads(pinned Threads issue resolved).  \n\nAuthor: Sven Loesekann\n\nTechnologies: Java, Kotlin, Spring Boot, Jpa, Postgresql, Soap, Rest, Kafka, Maven,\nKubernetes, Helm, Docker\n\n[![CodeQL](https://github.com/Angular2Guy/SimpleDataStreamPipelines/actions/workflows/codeql.yml/badge.svg)](https://github.com/Angular2Guy/SimpleDataStreamPipelines/actions/workflows/codeql.yml)\n\n## Articles\n\n[Simple Efficient Spring/Kafka DataStreams](https://angular2guy.wordpress.com/2025/02/24/simple-efficient-spring-kafka-datastreams/)\n\n[Using KRaft Kafka for development and Kubernetes deployment](https://angular2guy.wordpress.com/2024/08/17/using-kraft-kafka-for-development-and-kubernetes-deployment/)\n\n## Features\n\n1. SourceSink Application to create changes/events/requests to be processed by the DataStreams implemented in Kotlin.\n2. DatabaseToRest DataStream that uses Debezium to receive the changes in a table and send them to a rest endpoint.\n3. SoapToDb DataStream that receives Soap requests and stores them in a database with Jpa.\n4. EventToFile DataStream that receives Kafka events and stores them in files on the filesystem.\n5. A Helm chart to deploy Kafka(KRaft), Postgresql and all the applications of the system to a Minikube Kubernetes cluster.\n\n## Mission Statement\n\nThe project shows howto create and run DataStreams based on Jdk 24+. DataStreams are \na single application that contains the data source that receives the requests/events/changes \nand turns them into Kafka events and sends them to Kafka. The sink of the DataStream \nreceives the Kafka events and sends/stores them in the target system. The Virtual Threads \nmake efficient use of the system resources and the single application saves memory \nbecause of the shared JVM. To show this architecture this SourceSink Spring Boot application\ncreates database changes/Soap requests/Kafka events to trigger the DataStreams and receives \nRest calls of one DataStream. The SimpleDataStreamPipelines project has a Maven build\nto build all of the applications. \n\nFor deployment the SimpleDataStreamPipelines provides Dockerfiles that are used to \ncreate Docker images that can be deployed in a Minikube Kubernetes cluster. To \nconfigure/deploy the applications in the Docker images a Helm chart is provided. \n\nThis project has been created to show howto create simple efficient DataStreams. \nThe goal is to reduce complexity and use code instead of configuration. To do that a \nlittle bit of code duplication was accepted to enable the simplicity. The architecture \nis easy to understand and because of that easy to extend. \n\n## Setup\n\nFor development Intellij CE was used because it supports Java/Kotlin very well. \nTo run it for development the files 'runKafka.sh' and 'runPostgresql.sh' can be used \nto run the both as local Docker images. \n\nTo create the DataStreams the commands in the 'buildDocker.sh' files in the directories\ncan be used. The images need to be pushed to the DockerHub and then can be used with the \nHelm Chart in Minikube.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangular2guy%2Fsimpledatastreampipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fangular2guy%2Fsimpledatastreampipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangular2guy%2Fsimpledatastreampipelines/lists"}