{"id":24951924,"url":"https://github.com/romanow/data-migration-job","last_synced_at":"2025-03-28T19:40:24.567Z","repository":{"id":275336378,"uuid":"847698682","full_name":"Romanow/data-migration-job","owner":"Romanow","description":"Batch job for data migration between databases","archived":false,"fork":false,"pushed_at":"2025-02-19T10:46:15.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-19T11:25:42.855Z","etag":null,"topics":["database-migration","kubernetes-job","spring-batch"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Romanow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-26T11:27:12.000Z","updated_at":"2025-02-19T10:30:14.000Z","dependencies_parsed_at":"2025-02-01T19:31:42.384Z","dependency_job_id":"57ef7b42-7f54-405b-9e70-c41e158176fd","html_url":"https://github.com/Romanow/data-migration-job","commit_stats":null,"previous_names":["romanow/data-migration-job"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fdata-migration-job","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fdata-migration-job/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fdata-migration-job/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Romanow%2Fdata-migration-job/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Romanow","download_url":"https://codeload.github.com/Romanow/data-migration-job/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246091178,"owners_count":20722168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-migration","kubernetes-job","spring-batch"],"created_at":"2025-02-03T01:32:25.263Z","updated_at":"2025-03-28T19:40:24.562Z","avatar_url":"https://github.com/Romanow.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/Romanow/data-migration-job/actions/workflows/build.yml/badge.svg?branch=master)](https://github.com/Romanow/data-migration-job/actions/workflows/build.yml)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)\n[![Release](https://img.shields.io/github/v/release/Romanow/data-migration-job?logo=github\u0026sort=semver)](https://github.com/Romanow/data-migration-job/releases/latest)\n[![Docker Pulls](https://img.shields.io/docker/pulls/romanowalex/data-migration-job?logo=docker)](https://hub.docker.com/r/romanowalex/data-migration-job)\n[![License](https://img.shields.io/github/license/Romanow/data-migration-job)](https://github.com/Romanow/data-migration-job/blob/master/LICENSE)\n\n# Batch process for data migration\n\nGitHub: [romanow/data-migration-job](https://github.com/Romanow/data-migration-job).\n\n## Локальный запуск\n\nИспользуем [docker-compose.yml](docker-compose.yml)\n\n## Конфигурация\n\nСкрипт представляет собой приложение командной строки и запускается как k8s Job. Подключается к двум БД (`SOURCE_*` и\n`TARGET_*`) и поочередно перекладывает данные между таблицами, описанными в переменной `batch-processing.tables`. Пример\nописания переменной `batch-processing.tables`:\n\n```yaml\nbatch-processing:\n  tables:\n    - key-column-name: uid\n      source:\n        schema: public\n        table: users\n      target:\n        schema: public\n        table: users\n```\n\nПараметры подключения к SOURCE и TARGET базам данных ([`application.yml`](src/main/resources/application.yml)):\n\n```yaml\nspring:\n  datasource:\n    source:\n      jdbc-url: jdbc:postgresql://${SOURCE_DATABASE_HOST:postgres}:${SOURCE_DATABASE_PORT:5432}/${SOURCE_DATABASE_NAME:source}\n      username: ${SOURCE_DATABASE_USER:program}\n      password: ${SOURCE_DATABASE_PASSWORD:test}\n      driver-class-name: org.postgresql.Driver\n    target:\n      jdbc-url: jdbc:postgresql://${TARGET_DATABASE_HOST:postgres}:${TARGET_DATABASE_PORT:5432}/${TARGET_DATABASE_NAME:target}\n      username: ${TARGET_DATABASE_USER:program}\n      password: ${TARGET_DATABASE_PASSWORD:test}\n      driver-class-name: org.postgresql.Driver\n```\n\nДля подключения внешней конфигурации нужно положить файл [`/opt/config/external.yml`](external.yml).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fromanow%2Fdata-migration-job","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fromanow%2Fdata-migration-job","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fromanow%2Fdata-migration-job/lists"}