{"id":23380574,"url":"https://github.com/flix-tech/pyslicer","last_synced_at":"2025-04-08T07:43:02.369Z","repository":{"id":149086460,"uuid":"70576201","full_name":"flix-tech/pyslicer","owner":"flix-tech","description":null,"archived":false,"fork":false,"pushed_at":"2017-02-09T10:36:09.000Z","size":18,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-02-14T04:51:12.292Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flix-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-11T09:11:27.000Z","updated_at":"2016-12-13T18:52:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e0bdfac-927d-497f-82ca-ee8428f3f59d","html_url":"https://github.com/flix-tech/pyslicer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpyslicer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpyslicer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpyslicer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpyslicer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flix-tech","download_url":"https://codeload.github.com/flix-tech/pyslicer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247801198,"owners_count":20998331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-21T20:16:49.414Z","updated_at":"2025-04-08T07:43:02.359Z","avatar_url":"https://github.com/flix-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PySlicer v0.2\n\n### ⇒ wtf is that?\n\nScript to cut a piece of a big database in an accurate way.\n\n### ⇒ how it works\n\nIt reads the data (record by record) from a source database and puts it into\na target database. For that, it needs two connections for reading and for writing.\nConfiguration is defined in `config.yml` (check `config.dist.yml` for reference).\nUser for write connection should have DROP/CREATE/TRUNCATE privileges besides\nregular SELECT/INSERT. Data is copied according to set of rules defined in\n`schema.yml` (check `schema.dist.yml` for reference).\n\nTo copy data it runs several workers at once. Currently up to 8, one worker per table.\nEach worker opens own read and write connections. Redis connection is shared.\n\nWhen copying data it tries to maintain reference integrity by detecting primary\nand foreign keys and storing them in interim storage (which is Redis).\nAfter the whole volume of data copied it iterates of sets of foreign keys and copies missing records.\n\n### ⇒ command parameters reference\n\nRequired parameters are `read` and `write`, which are names of connections specified in `config.yml`.\n\nOptional:\n\n* `continue` - skip cleaning interim and target storage\n* `tables` - narrow copy procedure to a scope of tables (not that references\nto out-of-scope tables still will be copied)\n* `copy-schema` - runs mysqldump on source database and recreates target one\nfrom its output (table structure and routines, no triggers)\n\n### ⇒ how to use it\n\nYou'll need Python 3.2+ and pip for dependencies:\n\n```bash\n$ pip3 install -r requirements.txt\n$ python3 -u run.py --read=... --write=...\n$ mysqldump ... --routines --no-create-db --quick --skip-triggers | sed -E \"s/DEFINER=[^ ]+ //g\" \u003e /tmp/sliced_db.sql\n```\n\n### ⇒ roadmap\n\nIt's only v0.2, what's next? It needs a lot of stuff:\n\n* ~~parameters for Redis connection in configuration file~~\n* ~~option to set max number workers~~\n* ~~option to specify path to schema file~~\n* ~~support for several read connections (e.g. different slaves)~~\n* pretty names generator based on hash of source value\n* export and pack the resulting database\n* multi-threading implementation for reference traverse\n* fancy progress bar\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflix-tech%2Fpyslicer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflix-tech%2Fpyslicer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflix-tech%2Fpyslicer/lists"}