Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/oelmekki/datamith

Datamith is a MySQL database conversion/import/fusion framework.
https://github.com/oelmekki/datamith

Last synced: 17 days ago
JSON representation

Datamith is a MySQL database conversion/import/fusion framework.

Awesome Lists containing this project

README 

        
= Datamith



Datamith is a MySQL database conversion/import/fusion framework. To use it, you will have to define some rules in the tables/ directory.



A rule is a class derivated from Datamith::Converter. It can :



* simply transfert data

* convert charset ( using iconv )

* update data if a record already exists

* take the value of one field and load it in an other field

* do some basic type conversion between datetime and timestamp

* append data instead of preserving the primary key if append mode is set ( tweakable on a per record basis with a Proc conditionnal )

* remember the old primary value and the new one to keep track of the associations when in append mode



There is one rule file per table to convert, and each set of rules is lauched individually against each row. A rule file and the class it contains must be named accordingly. The filename can be prefixed with a number to set the order.



== Configuration and rake tasks



You should set the hosts, users, passwords and databases in config.yml.



As soon it is done, you're given some rake tasks to generate the rule files.



Generating a file for all the tables :



  rake tables:populate



Generating a file for a specific table : ( assuming there is an "users" table )



  rake tables:generate:users



You may also set in this file the "tables_to_convert" key. I should be a

list of the tables you want to be generated. The list order is preserved, so

rake tables:populate with :



  tables_to_convert: [ users, posts, comments ]



will generate :



  tables/10_users.rb

  tables/20_posts.rb

  tables/30_comments.rb



No list or empty list means all the tables.



== Execution



When you're done with your rules, simply launch :

  rake convert



If you prefer a dump as a sql file, you can use instead :

  rake convert:dump



( You are responsible of the output redirection )



== Conversion



You will find more informations in the documentation for Datamith::Converter .

Here is a example of what can be done :



  # tables/10_User.rb

  class User < Converter

    @@old_table = "xy_user"

    @@new_table = "users"

    @@old_primary_key = :account_number

    @@skip = %w( 1 2 )

    @@charset_from = "latin1"



    def run

      append :condition => Proc.new { |old,new| ( 15..20 ).include?( old[ :account_number ] ) }

      convert :int, :account_number, :id

      convert :string, :name

      convert :timestamp_to_datetime, :created_at

    end

  end

 

  # tables/20_Post.rb

  class Post < Converter

    @@old_table = "post"

    @@new_table = "posts"



    def run

      appended_FK :user_id, :user

      convert :int, :id

      convert :int, :user_id

      convert :string, :text

    end

  end

 

Here, the user table is converted in first place. The primary key of the old table, named `xy_user`, is `account_number`. It is

converted to `id`, the records with value 1 and 2 are skipped and those between 15 and 20 are appended as new record instead

of simply import them.



Then, the `post` table is proceeded. The `user_id` field refers to the id in `users`, so when the converter find a value between

15 and 20 ( which has been re-inserted ), it corrects it with the new id value. This is provided by the appended_FK() method.



For all the other records in `users` and for all the records in `posts` the converter check if the primary value exists in the new

table. If so, it updates the record ; else it inserts it.