{"id":13665709,"url":"https://github.com/hooopo/kiba-plus","last_synced_at":"2025-04-19T21:39:45.491Z","repository":{"id":56880092,"uuid":"58324365","full_name":"hooopo/kiba-plus","owner":"hooopo","description":"Kiba enhancement for Ruby ETL.","archived":false,"fork":false,"pushed_at":"2019-11-08T22:27:11.000Z","size":95,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-18T16:22:43.701Z","etag":null,"topics":["bulk","etl","kiba","mysql","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hooopo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-08T18:00:29.000Z","updated_at":"2022-11-02T02:25:15.000Z","dependencies_parsed_at":"2022-08-20T11:40:45.693Z","dependency_job_id":null,"html_url":"https://github.com/hooopo/kiba-plus","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooopo%2Fkiba-plus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooopo%2Fkiba-plus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooopo%2Fkiba-plus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooopo%2Fkiba-plus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hooopo","download_url":"https://codeload.github.com/hooopo/kiba-plus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232545218,"owners_count":18539620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk","etl","kiba","mysql","postgresql"],"created_at":"2024-08-02T06:00:48.138Z","updated_at":"2025-01-05T03:56:32.812Z","avatar_url":"https://github.com/hooopo.png","language":"Ruby","funding_links":[],"categories":["1. language"],"sub_categories":["1.1 ruby"],"readme":"# kiba-plus\nKiba enhancement for Ruby ETL. It connects to various data sources including relational, non-relational, and flat file, cloud services and HTTP resources. It has flexible load strategies including insert, bulk load and upsert.\n\n# Usage\n\n```ruby\n# /tmp/customer_mysql_to_pg.etl\n\nrequire 'kiba/plus'\n\nrequire 'kiba/plus/source/mysql'\nrequire 'kiba/plus/destination/pg_bulk2'\n\nSOURCE_URL = 'mysql://root@localhost/shopperplus'\n\nDEST_URL   = 'postgresql://hooopo@localhost:5432/crm2_dev'\n\nsource Kiba::Plus::Source::Mysql, { :connect_url =\u003e SOURCE_URL,\n                           :query =\u003e %Q{SELECT id, email, 'hooopo' AS first_name, 'Wang' AS last_name FROM customers}\n                         }\n\ndestination Kiba::Plus::Destination::PgBulk2, { :connect_url =\u003e DEST_URL,\n                                :table_name =\u003e \"customers\",\n                                :truncate =\u003e true,\n                                :columns =\u003e [:id, :email, :first_name, :last_name],\n                                :incremental =\u003e false\n                              }\n\npost_process do\n  result = PG.connect(DEST_URL).query(\"SELECT COUNT(*) AS num FROM customers\")\n  puts \"Insert total: #{result.first['num']}\"\nend\n```\n\nExecute in shell:\n\n```shell\n$ bundle exec kiba /tmp/customer_mysql_to_pg.etl\n\n# Output:\n# I, [2016-05-16T01:53:36.832565 #87909]  INFO -- : TRUNCATE TABLE customers;\n# I, [2016-05-16T01:53:36.841770 #87909]  INFO -- : COPY customers (id, email, first_name, last_name) FROM STDIN WITH DELIMITER ',' NULL '\\N' CSV\n# Insert total: 428972\n```\n\nExecute in ruby script:\n\n```ruby\nrequire 'kiba'\n\njob_definition = Kiba.parse(IO.read('/tmp/customer_mysql_to_pg.etl'), '/tmp/customer_mysql_to_pg.etl')\nKiba.run(job_definition)\n```\n\n# Examples\n\n* [CSV to MySQL](https://github.com/hooopo/kiba-plus/blob/master/examples/customer_csv_to_mysql.etl)\n* [CSV to PG](https://github.com/hooopo/kiba-plus/blob/master/examples/customer_csv_to_pg.etl)\n* [MySQL to CSV](https://github.com/hooopo/kiba-plus/blob/master/examples/customer_mysql_to_csv.etl)\n* [MySQL to PG](https://github.com/hooopo/kiba-plus/blob/master/examples/customer_mysql_to_pg.etl)\n* [MySQL incremental to PG](https://github.com/hooopo/kiba-plus/blob/master/examples/incremental_insert.etl)\n\n# Main Feature\n\n* Csv Source\n* MySQL Source\n* Postgresql Source\n* Citus Source\n* Greenplus Source\n* MongoDB Source (TODO)\n* Elastic Source (TODO)\n* Redshift Source (TODO)\n\n* Csv Destination\n* MySQL Destination\n* Postgresql Destination\n* Citus Destination\n* Greenplus Destination\n* MongoDB Destination (TODO)\n* Elastic Destination (TODO)\n* Redshift Destination (TODO)\n\n* Bulk Load for large dataset\n* Upsert for MySQL \u0026 Postgresql\n* Incremental Update\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'kiba'\ngem 'kiba-plus'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install kiba-plus\n\n## Development\n\nFirst of all, Please run the following code in shell.\n\n```bash\n\n$ mysql -e 'create database kiba_plus_src_test;'\n\n$ mysql -e 'create database kiba_plus_dest_test;'\n\n$ psql -c 'create database kiba_plus_src_test;' -U postgres\n\n$ psql -c 'create database kiba_plus_dest_test;' -U postgres\n\n```\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/hooopo/kiba-plus.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhooopo%2Fkiba-plus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhooopo%2Fkiba-plus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhooopo%2Fkiba-plus/lists"}