{"id":19766221,"url":"https://github.com/akaliutau/spring-batch","last_synced_at":"2025-02-28T04:16:04.985Z","repository":{"id":120056263,"uuid":"358161946","full_name":"akaliutau/spring-batch","owner":"akaliutau","description":"Asynchronous data processing on the basis of Spring Batch","archived":false,"fork":false,"pushed_at":"2021-04-20T15:26:13.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-11T00:13:31.699Z","etag":null,"topics":["spring-batch","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akaliutau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-15T07:08:46.000Z","updated_at":"2021-04-20T15:26:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"9814ab7e-5618-4476-bf1c-2ccd460f8927","html_url":"https://github.com/akaliutau/spring-batch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fspring-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fspring-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fspring-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fspring-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akaliutau","download_url":"https://codeload.github.com/akaliutau/spring-batch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241099623,"owners_count":19909577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["spring-batch","spring-boot"],"created_at":"2024-11-12T04:23:27.189Z","updated_at":"2025-02-28T04:16:04.969Z","avatar_url":"https://github.com/akaliutau.png","language":"Java","readme":"About\n======\n\nThis is a simple project implementing asynchronous data processing on the basis of Spring Batch.\n\nOverview\n=========\n\nWe are creating a job to perform reading line by line csv file, preparing insert statement and executing it on the database.\n\nThe whole workflow is defined in two files: BatchIO and JobFactory.\n\nHere is a brief description of some common classes:\n\n1. JobBuilderFactory(JobRepository jobRepository)  - a convenient factory for a JobBuilder to build a job\n\n2. StepBuilderFactory which sets the JobRepository - a convenient factory to build steps.\n\nA Step is a domain object that contains an independent, sequential phase of a batch job and contains all of the information needed to define and control the actual batch processing. \n\n\nNow that we’ve created the reader and processor for data we need to write it. For the reading, we’ve been using chunk-oriented processing, meaning we’ve been reading the data one at a time.\n\n3. FlatFileItemReader\u003cT\u003e ItemReader that reads lines from input setResource(Resource) In this project we are using CSV file.\n\nFlatFileItemReader can be parameterized:\n* setLineMapper method converts Strings to objects representing the item.\n* setResource. Public setter for the input resource.\n* setLinesToSkip the number of lines to skip at the start of a file.\n\n4. Defining the Datasource DataSource dataSource().\n\nHere we can define the type of datasource i.e (MySql, Oracle etc) and scripts specific to the datasource are defined. For Spring Boot it is enough to provide a dependency in pom file and schema.\n\n5. The JdbcBatchItemWriter.\n\nUsed to integrate the datasource and the itemWriter object - it will set JDBC connection and the sql statement to execute in the database.\n\n\nNotes\n======\n\nThe most tricky moment is tied with correct shutdown of Spring Batch job schedulers. The issue is - when other JDBC sources are used along with JPA this results in live threads after all jobs have completed their work.\n\nAs for Spring Boot 2.4.x it is still the unresolved issue.\n\nIn this project the problem is solved through introduction of ShutdownBatchHook singleton class which is used to track the state of all jobs and to invoke explicitly close() method on ConfigurableApplicationContext. \n\n\nBuilding project\n=================\n\n```\nmvn clean package\n``` \n\nRun as the standard Spring Boot app:\n\n```\njava -jar spring-batch-csv-0.0.1.jar\n```\n\nObserve the output as the following one:\n\n```\n34.515  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : Starting SpringBatchCsvApplication v0.0.1 using Java 14.0.1\n34.515  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : No active profile set, falling back to default profiles: default\n35.228  INFO 4024 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.\n35.244  INFO 4024 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 10 ms. Found 0 JPA repository interfaces.\n35.653  INFO 4024 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...\n36.048  INFO 4024 --- [           main] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)\n36.048  INFO 4024 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.\n36.190  INFO 4024 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]\n36.299  INFO 4024 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.4.25.Final\n36.534  INFO 4024 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}\n36.705  INFO 4024 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect\n36.943  INFO 4024 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]\n36.958  INFO 4024 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'\n36.974  WARN 4024 --- [           main] o.s.b.c.c.a.DefaultBatchConfigurer       : No transaction manager was provided, using a DataSourceTransactionManager\n36.990  INFO 4024 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL\n37.194  INFO 4024 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.\n37.225  INFO 4024 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'threadPoolTaskExecutor'\n37.429  INFO 4024 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'\n37.475  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : Started SpringBatchCsvApplication in 3.416 seconds (JVM running for 3.907)\n37.475  INFO 4024 --- [           main] o.s.b.a.b.JobLauncherApplicationRunner   : Running default command line with: []\n37.475  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : EXECUTING : command line runner\n37.615  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : csvFileToDatabaseJob\n37.615  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : 0\n37.615  INFO 4024 --- [lTaskExecutor-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=csvFileToDatabaseJob]] launched with the following parameters: [{name=job 0}]\n37.615  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : APPLICATION FINISHED\n37.615  INFO 4024 --- [           main] org.springboot.config.ShutdownBatchHook  : initiate wait loop\n37.662  INFO 4024 --- [lTaskExecutor-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [csvFileToDatabaseStep]\n37.709  INFO 4024 --- [lTaskExecutor-1] org.springboot.processor.MovieProcessor  : Converting MovieInfo [id=1, title=Cowboy Bebop, description=In the year 2071, humanity has colonized ...] -\u003e MovieInfo [id=1, title=Cowboy Bebop, description=In the year 2071, humanity has colonized ...]\n37.724  INFO 4024 --- [lTaskExecutor-1] org.springboot.processor.MovieProcessor  : Converting MovieInfo [id=1000, title=Uchuu Kaizoku Captain Harlock, description=The year is 2977. Mankind has become com ...] -\u003e MovieInfo [id=1000, title=Uchuu Kaizoku Captain Harlock, description=The year is 2977. Mankind has become com ...]\n37.724  INFO 4024 --- [lTaskExecutor-1] org.springboot.processor.MovieProcessor  : Converting MovieInfo [id=10012, title=Carnival Phantasm, description=How do you resolve a conflict between po ...] -\u003e MovieInfo [id=10012, title=Carnival Phantasm, description=How do you resolve a conflict between po ...]\n37.740  INFO 4024 --- [lTaskExecutor-1] org.springboot.processor.MovieProcessor  : Converting MovieInfo [id=9989, title=Ano Hi Mita Hana no Namae wo Bokutachi wa Mada Shiranai., description=After a tragic accident during their chi ...] -\u003e MovieInfo [id=9989, title=Ano Hi Mita Hana no Namae wo Bokutachi wa Mada Shiranai., description=After a tragic accident during their chi ...]\n37.740  INFO 4024 --- [lTaskExecutor-1] org.springboot.processor.MovieProcessor  : Converting MovieInfo [id=9996, title=Hyouge Mono, description=The story is set during Japan's Sengoku  ...] -\u003e MovieInfo [id=9996, title=Hyouge Mono, description=The story is set during Japan's Sengoku  ...]\n37.755  INFO 4024 --- [lTaskExecutor-1] o.s.batch.core.step.AbstractStep         : Step: [csvFileToDatabaseStep] executed in 93ms\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : ============ JOB FINISHED ============\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Verifying the results...\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Found MovieInfo [id=1, title=Cowboy Bebop, description=In the year 2071, humanity has colonized ...] in the db\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Found MovieInfo [id=1000, title=Uchuu Kaizoku Captain Harlock, description=The year is 2977. Mankind has become com ...] in the db\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Found MovieInfo [id=10012, title=Carnival Phantasm, description=How do you resolve a conflict between po ...] in the db\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Found MovieInfo [id=9989, title=Ano Hi Mita Hana no Namae wo Bokutachi wa Mada Shiranai., description=After a tragic accident during their chi ...] in the db\n37.755  INFO 4024 --- [lTaskExecutor-1] .s.b.l.JobCompletionNotificationListener : Found MovieInfo [id=9996, title=Hyouge Mono, description=The story is set during Japan's Sengoku  ...] in the db\n37.755  INFO 4024 --- [lTaskExecutor-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=csvFileToDatabaseJob]] completed with the following parameters: [{name=job 0}] and the following status: [COMPLETED] in 109ms\n40.627  INFO 4024 --- [           main] org.springboot.config.ShutdownBatchHook  : all jobs finished {0=COMPLETED}\n40.627  INFO 4024 --- [           main] org.springboot.config.ShutdownBatchHook  : closing applicationContext\n40.627  INFO 4024 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'taskScheduler'\n40.627  INFO 4024 --- [           main] o.springboot.SpringBatchCsvApplication   : gracefull shutdown\n40.627  INFO 4024 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'threadPoolTaskExecutor'\n40.627  INFO 4024 --- [           main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'\n40.627  INFO 4024 --- [           main] .SchemaDropperImpl$DelayedDropActionImpl : HHH000477: Starting delayed evictData of schema as part of SessionFactory shut-down'\n40.627  INFO 4024 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...\n40.643  INFO 4024 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.\n\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakaliutau%2Fspring-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakaliutau%2Fspring-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakaliutau%2Fspring-batch/lists"}