{"id":20568422,"url":"https://github.com/ochaton/migrate","last_synced_at":"2025-09-17T16:48:13.002Z","repository":{"id":47795964,"uuid":"320078205","full_name":"ochaton/migrate","owner":"ochaton","description":"Library to perform live migration of data from Tarantool 1.5 to modern Tarantool","archived":false,"fork":false,"pushed_at":"2022-08-23T16:00:17.000Z","size":57,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-16T20:44:50.596Z","etag":null,"topics":["migrate","tarantool"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ochaton.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-09T20:59:35.000Z","updated_at":"2024-11-20T07:25:46.000Z","dependencies_parsed_at":"2023-01-16T17:00:16.395Z","dependency_job_id":null,"html_url":"https://github.com/ochaton/migrate","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ochaton%2Fmigrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ochaton%2Fmigrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ochaton%2Fmigrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ochaton%2Fmigrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ochaton","download_url":"https://codeload.github.com/ochaton/migrate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242179871,"owners_count":20085051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["migrate","tarantool"],"created_at":"2024-11-16T04:50:15.588Z","updated_at":"2025-09-17T16:48:07.931Z","avatar_url":"https://github.com/ochaton.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Linter check](https://github.com/ochaton/migrate/actions/workflows/lint.yml/badge.svg)\n![Release status](https://github.com/ochaton/migrate/actions/workflows/push-rockspec.yml/badge.svg)\n\n# Migrate\nRead Tarantool 1.5 snapshots, xlogs and replication from modern Tarantool.\n\nThis module aims to ease migration from Tarantool 1.5 to modern versions of Tarantool (1.10.14+).\n\n## Status\n\nTested in production\n\n## Version\n\nThis document describes migrate 0.1.0\n\n## Installation\n* luarocks \u003e= 2.4.2\n* tarantool \u003e= 1.10\n\n```bash\nluarocks --tree .rocks --lua-version 5.1 --server http://moonlibs.github.io/rocks \\\n    install https://github.com/ochaton/migrate/releases/latest/download/migrate-scm-1.src.rock\n\n# Install tarantool from releases\ntarantoolctl rocks --server=https://moonlibs.github.io/rocks install https://github.com/ochaton/migrate/releases/latest/download/migrate-scm-1.src.rock\n```\n\nThis library also published to https://moonlibs.github.io/rocks\n\nStarting from Tarantool 2.10.0 you may add url https://moonlibs.github.io/rocks to you `.rocks/config-5.1.lua` and install library like this:\n\n```bash\ntarantoolctl rocks install migrate\n```\n\nConfiguration of rocks servers should be following:\n```\n$ cat .rocks/config-5.1.lua\nrocks_servers = {\n        \"http://moonlibs.github.io/rocks\",        -- moonlibs libs\n        \"http://rocks.tarantool.org/\",            -- tarantool libs\n        \"http://luarocks.org/repositories/rocks\", -- luarocks.org libs\n}\n```\n\n## Synopsis\n\nThis library provides luafun interface to read binary snapshots, xlogs and replication.\n\n```lua\n-- Read single snapshot\nrequire \"migrate\".pairs \"/path/to/00000000001342537118.snap\":length()\n\n-- Read directory with snapshots and xlogs\nrequire \"migrate\".paris \"/path/to/snap_and_xlog_dir\":length()\nrequire \"migrate\".pairs { dir = \"/path/to/snap_and_xlog_dir\" }:length()\n\n-- Read snapshots and xlogs from different directories\nrequire \"migrate\".pairs { dir = { snap = \"/path/to/snaps\", xlog = \"/path/to/xlogs\" } }:length()\n\n-- Read only xlogs (will read all xlogs from LSN=0. May raise if no xlog found)\nrequire \"migrate\".pairs { dir = { xlog = \"/path/to/xlogs\" } }:length()\n\n-- Read xlogs from specified LSN (all xlogs gaps will be checked)\nrequire \"migrate\".pairs { dir = { xlog = \"/path/to/xlogs\" }, confirmed_lsn = 10 }:length()\n\n-- Read xlogs from specified LSN (without LSN Gaps check)\nrequire \"migrate\".pairs { dir = { xlog = \"/path/to/xlogs\" }, confirmed_lsn = 10, checklsn = false }:length()\n\n-- Read transactions from replication (without LSN checks)\nrequire \"migrate\".replica{ host = \"tarantool.host.i\", port = 3301, confirmed_lsn = 0 }:take(100):each(require'log'.info)\n\n-- Execute full pipeline with bootstraping from snaps, xlogs and replication\nrequire \"migrate\".pairs {\n\tdir = {\n\t\txlog = \"/path/to/xlogs\",\n\t\tsnap = \"/path/to/snaps\",\n\t},\n\t-- persist creates space and persists lsn of each transactions\n\t-- in space box.space._migration\n\t-- enabling persist forces each tuple to be processed in separate transaction\n\tpersist = true, -- (default false)\n\t-- txn allows to configure how many xlogs will share same transaction\n\t-- txn takes effect only with persist\n\ttxn = 1, -- (default 1)\n\treplication = {\n\t\thost = \"legacy-master.addr.i\",\n\t\tport = 3301, -- primary port,\n\t\ttimeout = 1.5,\n\t\treconnect = 1/3, -- default in seconds\n\t\tdebug = false,\n\t},\n}\n-- Iterator will gracefully finished when all snaps and xlogs were processed.\n-- If replication is enabled iterator will finish when it confirms read_only master's lsn.\n```\n\n## Example\nNaive example\n\n```lua\n-- default log_level=5 (info), log_level=6 (verbose) gives you more information about migration process\nbox.cfg{log_level = 6}\n\nrequire \"migrate\".pairs {\n\tdir = \"/path/to/snaps_and_xlogs\",\n\tpersist = true,\n\ttxn = 1000,\n\tdebug = false, -- enable this if you want more information about migration process\n\treplication = {\n\t\thost = \"legacy-master.addr.i\",\n\t\tport = 3301, -- primary port, replica port will be discovered automatically\n\t\ttimeout = 3, -- write timeout in seconds\n\t\treconnect = 1/3, -- default in seconds\n\t\tdebug = false, -- enable this if you want more verbose information about replication\n\t}\n}:each(function(t)\n\tlocal h = t.HEADER\n\tlocal space_no = t.BODY.space_id\n\tlocal tuple = t.BODY.tuple\n\tlocal key = t.BODY.key\n\tlocal op = t.HEADER.type -- UPDATE, INSERT, DELETE or REPLACE\n\n\tlocal tuple = migrate_tuple(space_no, tuple)\n\tif key then\n\t\tkey = migrate_primary(space_no, key)\n\tend\n\n\tlocal space = box.space[512+space_no]\n\tif op == 'INSERT' then\n\t\tspace:insert(tuple)\n\telseif op == 'DELETE' then\n\t\tspace:delete(key)\n\telseif op == 'UPDATE' then\n\t\tspace:update(key, tuple)\n\telseif op == 'REPLACE' then\n\t\tspace:replace(tuple)\n\tend\nend)\n```\n\n## TODO\n* Tests and doc\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fochaton%2Fmigrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fochaton%2Fmigrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fochaton%2Fmigrate/lists"}