https://github.com/schubergphilis/data-migrator

A declarative data-migration package
https://github.com/schubergphilis/data-migrator

anonymization data-policy data-structures database development-tools development-utils gdpr kinesis migration python-library

Last synced: 19 days ago
JSON representation

A declarative data-migration package

• Host: GitHub
• URL: https://github.com/schubergphilis/data-migrator
• Owner: schubergphilis
• License: mit
• Created: 2017-04-13T14:50:29.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2024-10-29T22:51:53.000Z (about 2 months ago)
• Last Synced: 2024-10-30T00:47:48.738Z (about 2 months ago)
• Topics: anonymization, data-policy, data-structures, database, development-tools, development-utils, gdpr, kinesis, migration, python-library
• Language: Python
• Homepage: http://data-migrator.readthedocs.io/en/latest/
• Size: 273 KB
• Stars: 16
• Watchers: 6
• Forks: 4
• Open Issues: 17
• Metadata Files:
  • Readme: README.rst
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Authors: AUTHORS

Awesome Lists containing this project

README

        
Data-migrator (version 0.6.3.dev2) is a simple data-migration package for python lovers.

.. image:: https://img.shields.io/pypi/pyversions/data-migrator.svg

    :target: https://pypi.python.org/pypi/data-migrator

.. image:: https://circleci.com/gh/schubergphilis/data-migrator.svg?style=shield

    :target: https://circleci.com/gh/schubergphilis/data-migrator

----

.. image:: https://pyup.io/repos/github/schubergphilis/data-migrator/shield.svg

     :target: https://pyup.io/repos/github/schubergphilis/data-migrator/

     :alt: Updates

.. image:: https://readthedocs.org/projects/data-migrator/badge/?version=latest

    :target: http://data-migrator.readthedocs.io/en/latest/?badge=latest

    :alt: Documentation Status

.. image:: https://api.codacy.com/project/badge/Grade/bf6030e9e7e248979607802880336611

    :target: https://www.codacy.com/app/schubergphilis/data-migrator?utm_source=github.com&utm_medium=referral&utm_content=schubergphilis/data-migrator&utm_campaign=Badge_Grade

.. image:: https://api.codacy.com/project/badge/Coverage/bf6030e9e7e248979607802880336611

    :target: https://www.codacy.com/app/schubergphilis/data-migrator?utm_source=github.com&utm_medium=referral&utm_content=schubergphilis/data-migrator&utm_campaign=Badge_Coverage

.. image:: https://badge.fury.io/py/data-migrator.svg

    :target: https://badge.fury.io/py/data-migrator

Data-migrator is a declarative DSL for table driven data transformations, set up as

an open and extensive system. Use this to create data transformations for

changing databases as a result of changing code, initial loads to datalakes

(it contains a Kinesis provider)and more.

Example

-------

Data-migrator assumes data is extracted and loaded with client access.

.. code-block:: bash

	$ mysql source_db -E 'select id,a,b from table' -B  | python my_filter.py | mysql target_db

It than offers a wide range of primitives with default settings to build complex transformations

fast, readable and extendable

.. code-block:: python

  from data_migrator import models, transform

  class Result(models.Model):

    id   = models.IntField(pos=0) # keep id

    uuid = models.UUIDField()     # generate new uuid4 field

    a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())

    b    = models.StringField(pos=2, name='my_b')

  if __name__ == "__main__":

    transform.Transformer(models=[Result]).process()

Installation

------------

Execute the following command to install *data-migrator* with ``pip``::

    pip install data-migrator

See the `Installation Instructions

`_ in Documentation for

more instructions on installing, upgrading, and uninstalling *data-migrator*.

The project is `maintained at GitHub `_.

Support and contribute

----------------------

Questions, comments, bug reports and especially tested patches may be

submitted directly to the `issue tracker

`_.

Everyone interacting with this codebase, issue trackers,

chat rooms, and mailing lists is expected to follow the

`Code of Conduct `_.