{"id":20645063,"url":"https://github.com/romanow/batch-migration","last_synced_at":"2026-04-19T05:37:30.205Z","repository":{"id":146005437,"uuid":"610445753","full_name":"Romanow/batch-migration","owner":"Romanow","description":"Large data migration with Spring Batch","archived":false,"fork":false,"pushed_at":"2023-03-17T08:18:39.000Z","size":1211,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-09T15:54:49.252Z","etag":null,"topics":["batch-processing","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Romanow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-06T19:40:53.000Z","updated_at":"2023-03-06T19:43:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"465de39b-4429-45b5-8f59-b2e2e101066e","html_url":"https://github.com/Romanow/batch-migration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Romanow/batch-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fbatch-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fbatch-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fbatch-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fbatch-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Romanow","download_url":"https://codeload.github.com/Romanow/batch-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fbatch-migration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266093086,"owners_count":23875548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","spring-boot"],"created_at":"2024-11-16T16:18:35.809Z","updated_at":"2026-04-19T05:37:30.156Z","avatar_url":"https://github.com/Romanow.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Batch Migration\n\n[![Build project](https://github.com/Romanow/batch-migration/actions/workflows/build.yml/badge.svg?branch=master)](https://github.com/Romanow/batch-migration/actions/workflows/build.yml)\n\n## Сборка и запуск\n\n```shell\n$ ./gradlew clean build\n$ docker compose build\n```\n\n### Run local k8s cluster\n\nЗапускаем локальный кластер k8s в docker (папка [`k8s`](k8s)):\n\n```shell\n$ kind create cluster --config kind.yml\n$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n```\n\nЗагружаем docker images в кластер `kind` (папка [`scripts`](scripts)):\n\n```shell\n$ ./scripts/load-images.sh\n```\n\nЗагружаем репозитории:\n\n```shell\n$ helm repo add romanow https://romanow.github.io/helm-charts/\n$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n$ helm repo update\n```\n\nУстанавливаем инструменты для мониторинга кластера:\n\n```shell\n$ helm install kube-state-metrics prometheus-community/kube-state-metrics --set image.tag=v2.6.0 \n\n$ helm install prometheus -f prometheus/values.yaml romanow/prometheus\n\n$ helm install grafana -f grafana/values.yaml romanow/grafana\n```\n\nЗапускаем `migration-service`:\n\n```shell\n$ helm install postgres romanow/postgres --values postgres/values.yaml      \n\n$ helm install migration-service romanow/java-service --values migration-service/values.yaml      \n```\n\n### Тестирование\n\n![Spring Batch Architecture](images/Spring%20Batch%20Architecture.png)\n\nПри параллельном выполнении в 10 потоков с блоками по 5000 записей на 2CPU, 4Gb:\n\n```\nStep: [migrate:partition9] executed in 3m29s368ms\nStep: [migrate:partition6] executed in 3m31s859ms\nStep: [migrate:partition7] executed in 3m35s659ms\nStep: [migrate:partition8] executed in 3m42s162ms\nStep: [migrate:partition0] executed in 3m42s171ms\nStep: [migrate:partition4] executed in 3m42s858ms\nStep: [migrate:partition2] executed in 3m44s359ms\nStep: [migrate:partition3] executed in 3m44s359ms\nStep: [migrate:partition5] executed in 3m44s587ms\nStep: [migrate:partition1] executed in 3m45s27ms\nStep: [migrate:manager] executed in 3m45s258ms\nExecuting step: [delete]\nRemoved 400000 items from staged\nStep: [delete] executed in 654ms\n\nJob: [SimpleJob: [name=migration]]\ncompleted with the following parameters:\n[\n  {\n    \"solveId\": {\n      \"value\": \"solve_id\",\n      \"type\": \"class java.lang.String\",\n      \"identifying\": true\n    },\n    \"key\": {\n      \"value\": \"2023-03-15T09:23:24.955656258\",\n      \"type\": \"class java.lang.String\",\n      \"identifying\": true\n    }\n  }\n]\nand the following status: [COMPLETED] in 4m6s649ms\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fromanow%2Fbatch-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fromanow%2Fbatch-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fromanow%2Fbatch-migration/lists"}