{"id":19922292,"url":"https://github.com/isopropylcyanide/springbatch-kafkadb","last_synced_at":"2025-07-17T01:35:38.957Z","repository":{"id":118644635,"uuid":"143862575","full_name":"isopropylcyanide/SpringBatch-KafkaDB","owner":"isopropylcyanide","description":"A small demo that leverages Spring batch's capabilities to do job processing and Apache Kafka's stream processing. A simple CSV file gets used up in a batch job which then writes it to a Kafka queue (and H2 database) for further processing. ","archived":false,"fork":false,"pushed_at":"2018-09-06T12:07:39.000Z","size":28,"stargazers_count":14,"open_issues_count":0,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-03T07:37:12.499Z","etag":null,"topics":["apache-kafka","h2-database","kafka","kafka-consumer","kafka-producer","spring-batch","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/isopropylcyanide.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-08-07T11:16:48.000Z","updated_at":"2025-01-09T07:09:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"ec2f98ba-a7b5-4fc7-9b85-003d2433a42a","html_url":"https://github.com/isopropylcyanide/SpringBatch-KafkaDB","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/isopropylcyanide/SpringBatch-KafkaDB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/isopropylcyanide%2FSpringBatch-KafkaDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/isopropylcyanide%2FSpringBatch-KafkaDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/isopropylcyanide%2FSpringBatch-KafkaDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/isopropylcyanide%2FSpringBatch-KafkaDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/isopropylcyanide","download_url":"https://codeload.github.com/isopropylcyanide/SpringBatch-KafkaDB/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/isopropylcyanide%2FSpringBatch-KafkaDB/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265558733,"owners_count":23787970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","h2-database","kafka","kafka-consumer","kafka-producer","spring-batch","spring-boot"],"created_at":"2024-11-12T22:10:29.296Z","updated_at":"2025-07-17T01:35:38.952Z","avatar_url":"https://github.com/isopropylcyanide.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"##  SpringBatch-KafkaDB Demo ##\n---\n\n### What is it? ###\nA small demo that leverages Spring batch's capabilities to do job processing and Apache Kafka's stream processing.\nA simple CSV file is used in a batch job which then writes it to a Kafka Producer for further processing.\nA Kafka consumer can then verify by consuming the messages from the correct topic.\n\n![image](https://user-images.githubusercontent.com/12872673/43775559-cf5ab356-9a6a-11e8-89e9-162d5acae872.png)\n\n  \n---\n\n\n### Spring Batch pipeline ###\nBelow mentioned pipeline has been followed through out the codebase. The implementation is trivial once you modularize the responsibilities of each relevant class. \n\n![image](https://user-images.githubusercontent.com/12872673/43773992-394a9156-9a65-11e8-93df-ceb3e95a5889.png)\n\n---\n\n\n### Prerequisites ###\n* Spring Boot + Batch + JPA  \n* Apache Kafka\n* Apache Zookeeper\n\n---\n\n### Expectation ###\n Batch systems offer tremendous advantages as compared to interactive systems.\n * Repeated jobs are done fast in batch systems without user interaction.\n * You don’t need special hardware and system support to input data in batch systems.\n * Best for large organizations but small organizations can also benefit from it.\n ---\n Expectation is to convert the following flat file into something meaningful when run as a batch process.\n \n ![image](https://user-images.githubusercontent.com/12872673/43774267-30bed9ba-9a66-11e8-88b1-19978d0e6397.png)\n \n * Such as a Kafka stream like this *\n \n ![image](https://user-images.githubusercontent.com/12872673/43774875-4646bd3c-9a68-11e8-9bcd-a352e8b36618.png)\n\n\n * Or to a datastore like this *\n \n ![image](https://user-images.githubusercontent.com/12872673/43774750-d6392048-9a67-11e8-8d8d-8a11db5750ef.png)\n\n---\n\n### Setting up Apache Kafka ###\n```\n  # Start Zookeeper instance \n  $ zookeeper-server-start.bat ..\\..\\config\\zookeeper.properties\n  \n  # Start Kafka server\n  $ kafka-server-start.bat ..\\..\\config\\server.properties\n  \n  # Create a topic\n  $ kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic CSV_TOPIC_K\n  \n```\n Make sure following is appended to **config\\server.properties**\n ```\n port = 9092\n advertised.host.name = localhost \n ```\n \n ---\n\n### What are the list of branches ###\nBranch | Description\n------------ | -------------\nmaster | Base branch that reads from CSV and processes them to a topic in a Kafka producer\nbatch-db-upload | Similar to master except that it deserializes the CSV to a H2 Database instead of Kafka\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fisopropylcyanide%2Fspringbatch-kafkadb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fisopropylcyanide%2Fspringbatch-kafkadb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fisopropylcyanide%2Fspringbatch-kafkadb/lists"}