{"id":15492297,"url":"https://github.com/stranger6667/xdump","last_synced_at":"2025-04-22T19:42:47.520Z","repository":{"id":55120595,"uuid":"114687637","full_name":"Stranger6667/xdump","owner":"Stranger6667","description":"A consistent partial database copy \u0026 load utility","archived":false,"fork":false,"pushed_at":"2023-01-17T14:24:57.000Z","size":194,"stargazers_count":18,"open_issues_count":11,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-19T10:44:49.532Z","etag":null,"topics":["cli","database","django","dump","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Stranger6667.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"Stranger6667"}},"created_at":"2017-12-18T21:05:25.000Z","updated_at":"2023-08-10T15:38:53.000Z","dependencies_parsed_at":"2023-01-22T05:22:55.545Z","dependency_job_id":null,"html_url":"https://github.com/Stranger6667/xdump","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stranger6667%2Fxdump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stranger6667%2Fxdump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stranger6667%2Fxdump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Stranger6667%2Fxdump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Stranger6667","download_url":"https://codeload.github.com/Stranger6667/xdump/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250310235,"owners_count":21409626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","database","django","dump","postgresql"],"created_at":"2024-10-02T08:00:08.481Z","updated_at":"2025-04-22T19:42:47.483Z","avatar_url":"https://github.com/Stranger6667.png","language":"Python","funding_links":["https://github.com/sponsors/Stranger6667"],"categories":[],"sub_categories":[],"readme":"XDump\n=====\n\n.. image:: https://travis-ci.org/Stranger6667/xdump.svg?branch=master\n   :target: https://travis-ci.org/Stranger6667/xdump\n   :alt: Build Status\n\n.. image:: https://codecov.io/github/Stranger6667/xdump/coverage.svg?branch=master\n   :target: https://codecov.io/github/Stranger6667/xdump?branch=master\n   :alt: Coverage Status\n\n.. image:: https://readthedocs.org/projects/xdump/badge/?version=stable\n   :target: http://xdump.readthedocs.io/en/stable/?badge=stable\n   :alt: Documentation Status\n\n.. image:: https://img.shields.io/pypi/v/xdump.svg\n    :target: https://pypi.python.org/pypi/xdump\n    :alt: Latest PyPI version\n\nXDump is a utility to make a consistent partial dump and load it into the database.\n\nThe idea is to provide an ability to specify what to include in the dump via SQL queries.\n\nInstallation\n============\n\nXDump can be obtained with ``pip``::\n\n    $ pip install xdump\n\nUsage example\n=============\n\nMake a dump (on production replica for example):\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from xdump.postgresql import PostgreSQLBackend\n    \u003e\u003e\u003e\n    \u003e\u003e\u003e backend = PostgreSQLBackend(dbname='app_db', user='prod', password='pass', host='127.0.0.1', port='5432')\n    \u003e\u003e\u003e backend.dump(\n        '/path/to/dump.zip',\n        full_tables=['groups'],\n        partial_tables={'employees': 'SELECT * FROM employees ORDER BY id DESC LIMIT 2'}\n    )\n\nLoad a dump on your local machine:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e backend = PostgreSQLBackend(dbname='app_db', user='local', password='pass', host='127.0.0.1', port='5432')\n    # If you need a clear DB\n    \u003e\u003e\u003e backend.recreate_database()  # or `backend.truncate()`\n    \u003e\u003e\u003e backend.load('/path/to/dump.zip')\n\n\nDump is compressed by default. Compression level could be changed with passing ``compression`` argument to ``dump`` method.\nValid options are ``zipfile.ZIP_STORED``, ``zipfile.ZIP_DEFLATED``, ``zipfile.ZIP_BZIP2`` and ``zipfile.ZIP_LZMA``.\n\nThe verbosity of the output could be customized via ``verbosity`` (with values 0, 1 or 2) argument of a backend class.\n\nThere are two options to control the content of the dump:\n\n- ``dump_schema`` - controls if the schema should be included\n- ``dump_data`` - controls if the data should be included\n\nAutomatic selection of related objects\n++++++++++++++++++++++++++++++++++++++\n\nYou don't have to specify all queries for related objects - XDump will load them for you automatically. It covers\nboth, recursive and non-recursive relations.\nFor example, if the ``employees`` table has foreign keys ``group_id`` (to ``groups`` table) and ``manager_id``\n(to ``employees`` table) the resulting dump will have all objects related to selected employees\n(as well as for objects related to related objects, recursively).\n\nCommand Line Interface\n======================\n\n``xload`` provides an ability to create a dump.\n\nSignature:\n\n.. code-block:: bash\n\n    xdump [postgres|sqlite] [OPTIONS]\n\nCommon options::\n\n  -o, --output TEXT               output file name  [required]\n  -f, --full TEXT                 table name to be fully dumped. Could be used\n                                  multiple times\n  -p, --partial TEXT              partial tables specification in a form\n                                  \"table_name:select SQL\". Could be used\n                                  multiple times\n  -c, --compression [deflated|stored|bzip2|lzma]\n                                  dump compression level\n  --schema / --no-schema          include / exclude the schema from the dump\n  --data / --no-data              include / exclude the data from the dump\n  -D, --dbname TEXT               database to work with  [required]\n  -v, --verbosity                 verbosity level\n\nPostgreSQL-specific options::\n\n  -U, --user TEXT                 connect as specified database user\n                                  [required]\n  -W, --password TEXT             password for the DB connection\n  -H, --host TEXT                 database server host or socket directory\n  -P, --port TEXT                 database server port number\n\n``xload`` loads a dump into a database.\n\nSignature:\n\n\n.. code-block:: bash\n\n    xload [postgres|sqlite] [OPTIONS]\n\nCommon options::\n\n  -i, --input TEXT                input file name  [required]\n  -m, --cleanup-method [recreate|truncate]\n                                  method of DB cleaning up\n  -D, --dbname TEXT               database to work with  [required]\n  -v, --verbosity                 verbosity level\n\nPostgreSQL-specific options are the same as for ``xdump``.\n\nRDBMS support\n=============\n\nAt the moment only the following are supported:\n\n- PostgreSQL\n- SQLite \u003e= 3.8.3\n\nDjango support\n==============\n\nAdd ``xdump.extra.django`` to your ``INSTALLED_APPS`` settings:\n\n.. code-block:: python\n\n    INSTALLED_APPS = [\n       ...,\n       'xdump.extra.django',\n    ]\n\nAdd ``XDUMP`` to your project settings file. It should contain minimum two entries:\n\n- FULL_TABLES - a list of tables that should be fully dumped.\n- PARTIAL_TABLES - a dictionary with ``table_name``: ``select SQL``\n\n.. code-block:: python\n\n    XDUMP = {\n        'FULL_TABLES': ['groups'],\n        'PARTIAL_TABLES': {'employees': 'SELECT * FROM employees WHERE id \u003e 100'}\n    }\n\n\nOptionally you could use a custom backend:\n\n.. code-block:: python\n\n    XDUMP = {\n        ...,\n        'BACKEND': 'importable.string',\n    }\n\n\nRun ``xdump`` command::\n\n    $ ./manage.py xdump dump.zip\n\n\nRun ``xload`` command::\n\n    $ ./manage.py xload dump.zip\n\nPossible options to both commands:\n\n- ``-a/--alias`` - allows you to choose database config from ``DATABASES``, that is used during the execution;\n- ``-b/--backend`` - importable string, that leads to custom dump backend class.\n\nOptions for ``xdump`` command:\n\n- ``-s/--dump-schema`` - controls if the schema should be included;\n- ``-d/--dump-data`` - controls if the data should be included.\n\nOptions for ``xload`` command:\n\n- ``-m/--cleanup-method`` - optionally re-creates DB or truncates the data.\n\n**NOTE**. If the dump has no schema inside, DB won't be re-created.\n\nThe following ``make`` command could be useful to get a configured dump from production to your local machine:\n\n.. code-block:: bash\n\n    sync-production:\n        ssh -t $(TARGET) \"DJANGO_SETTINGS_MODULE=settings.production /path/to/manage.py xdump /tmp/dump.zip\"\n        scp $(TARGET):/tmp/dump.zip ./dump.zip\n        ssh -t $(TARGET) \"rm /tmp/dump.zip\"\n        DJANGO_SETTINGS_MODULE=settings.local $(PYTHON) manage.py xload ./dump.zip\n\nAnd the usage is:\n\n.. code-block:: bash\n\n    $ make sync-production TARGET=john@production.com PYTHON=/path/to/python/in/venv\n\nPython support\n==============\n\nXDump supports Python 2.7, 3.4 - 3.7 and PyPy 2 \u0026 3.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstranger6667%2Fxdump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstranger6667%2Fxdump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstranger6667%2Fxdump/lists"}