{"id":13879843,"url":"https://github.com/evilmartians/evil-seed","last_synced_at":"2025-05-15T17:01:39.328Z","repository":{"id":20567677,"uuid":"88243668","full_name":"evilmartians/evil-seed","owner":"evilmartians","description":"A Gem for creating partial anonymized dumps of your database using your app model relations.","archived":false,"fork":false,"pushed_at":"2025-02-20T06:36:52.000Z","size":150,"stargazers_count":532,"open_issues_count":3,"forks_count":21,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-03-31T21:46:39.367Z","etag":null,"topics":["activerecord","database","database-dump","mysql","postgresql","rails","replication","ruby","seeds"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evilmartians.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-14T07:20:07.000Z","updated_at":"2025-03-26T19:32:48.000Z","dependencies_parsed_at":"2024-06-18T13:03:39.198Z","dependency_job_id":"69383e8c-4a2e-499a-9d5b-4ff99f8a60ae","html_url":"https://github.com/evilmartians/evil-seed","commit_stats":{"total_commits":60,"total_committers":8,"mean_commits":7.5,"dds":"0.19999999999999996","last_synced_commit":"513f1673c04bb3ee4db5750beb88e605a0381532"},"previous_names":["palkan/evil-seed"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilmartians%2Fevil-seed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilmartians%2Fevil-seed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilmartians%2Fevil-seed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilmartians%2Fevil-seed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evilmartians","download_url":"https://codeload.github.com/evilmartians/evil-seed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247737788,"owners_count":20987721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","database","database-dump","mysql","postgresql","rails","replication","ruby","seeds"],"created_at":"2024-08-06T08:02:35.485Z","updated_at":"2025-04-07T22:07:13.464Z","avatar_url":"https://github.com/evilmartians.png","language":"Ruby","funding_links":[],"categories":["Ruby","Gems"],"sub_categories":["Data Management"],"readme":"[![Gem Version](https://badge.fury.io/rb/evil-seed.svg)](https://rubygems.org/gems/evil-seed)\n[![Build Status](https://travis-ci.org/evilmartians/evil-seed.svg?branch=master)](https://travis-ci.org/evilmartians/evil-seed)\n[![Cult of Martians](http://cultofmartians.com/assets/badges/badge.svg)](http://cultofmartians.com/tasks/evil-seed.html)\n\n# EvilSeed\n\nEvilSeed is a tool for creating partial anonymized dump of your database based on your app models.\n\n\u003ca href=\"https://evilmartians.com/\"\u003e\n\u003cimg src=\"https://evilmartians.com/badges/sponsored-by-evil-martians.svg\" alt=\"Sponsored by Evil Martians\" width=\"236\" height=\"54\"\u003e\u003c/a\u003e\n\n## Motivation\n\nUsing production-like data in your staging environment could be very useful, especially for debugging intricate production bugs.\n\nThe easiest way to achieve this is to use production database backups. But that's not an option for rather large applications for two reasons: \n\n- production dump can be extremely large, and it just can't be dumped and restored in a reasonable time\n\n- you should care about sensitive data (anonymization).\n\nEvilSeed aims to solve these problems.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'evil-seed', require: false\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install evil-seed\n\n## Usage\n\n### Configuration\n\n```ruby\nrequire 'evil_seed'\n\nEvilSeed.configure do |config|\n  # First, you should specify +root models+ and their +constraints+ to limit the number of dumped records:\n  # This is like Forum.where(featured: true).all\n  config.root('Forum', featured: true) do |root|\n    # You can limit number of records to be dumped\n    root.limit(100)\n    # Specify order for records to be selected for dump\n    root.order(created_at: :desc)\n\n    # It's possible to remove some associations from dumping with pattern of association path to exclude\n    #\n    # Association path is a dot-delimited string of association chain starting from model itself:\n    # example: \"forum.users.questions\"\n    root.exclude(/\\btracking_pixels\\b/, 'forum.popular_questions', /\\Aforum\\.parent\\b/)\n\n    # Include back only certain association chains\n    root.include(parent: {questions: %i[answers votes]})\n    # which is the same as\n    root.include(/\\Aforum(\\.parent(\\.questions(\\.(answers|votes))?)?)?\\z/)\n\n    # You can also specify custom scoping for associations\n    root.include(questions: { answers: :reactions }) do\n      order(created_at: :desc) # Any ActiveRecord query method is allowed\n    end\n\n    # It's possible to limit the number of included into dump has_many and has_one records for every association\n    # Note that belongs_to records for all not excluded associations are always dumped to keep referential integrity.\n    root.limit_associations_size(100)\n\n    # Or for certain association only\n    root.limit_associations_size(5, 'forum.questions')\n    root.limit_associations_size(15, 'forum.questions.answers')\n    # or\n    root.limit_associations_size(5, :questions)\n    root.limit_associations_size(15, questions: :answers)\n\n    # Limit the depth of associations to be dumped from the root level\n    # All traverses through has_many, belongs_to, etc are counted\n    # So forum.subforums.subforums.questions.answers will be 5 levels deep\n    root.limit_deep(10)\n  end\n\n  # Everything you can pass to +where+ method will work as constraints:\n  config.root('User', 'created_at \u003e ?', Time.current.beginning_of_day - 1.day)\n\n  # For some system-wide models you may omit constraints to dump all records\n  config.root(\"Role\") do |root|\n    # Exclude everything\n    root.exclude(/.*/)\n  end\n\n  # Transformations allows you to change dumped data e. g. to hide sensitive information\n  config.customize(\"User\") do |u|\n    # Reset password for all users to the same for ease of debugging on developer's machine\n    u[\"encrypted_password\"] = encrypt(\"qwerty\")\n    # Reset or mutate other attributes at your convenience\n    u[\"metadata\"].merge!(\"foo\" =\u003e \"bar\")\n    u[\"created_at\"] = Time.current\n    # Please note that there you have only hash of record attributes, not the record itself!\n  end\n\n  # Anonymization is a handy DSL for transformations allowing you to transform model attributes in declarative fashion\n  # Please note that model setters will NOT be called: results of the blocks will be assigned to\n  config.anonymize(\"User\") do\n    name  { Faker::Name.name }\n    email { Faker::Internet.email }\n    login { |login| \"#{login}-test\" }\n  end\n\n  # You can ignore columns for any model. This is specially useful when working\n  # with encrypted columns.\n  #\n  # This will remove the columns even if the model is not a root node and is\n  # dumped via an association.\n  config.ignore_columns(\"Profile\", :name)\n\n  # Disable foreign key nullification for records that are not included in the dump\n  # By default, EvilSeed will nullify foreign keys for records that are not included in the dump\n  config.dont_nullify = true\n\n  # Unscope relations to include soft-deleted records etc\n  # This is useful when you want to include all records, including those that are hidden by default\n  # By default, EvilSeed will abide default scope of models\n  config.unscoped = true\n\n  # Verbose mode will print out the progress of the dump to the console along with writing the file\n  # By default, verbose mode is off\n  config.verbose = true\n  config.verbose_sql = true\nend\n```\n\n### Creating dump\n\nJust call the `#dump` method and pass a path where you want your SQL dump file to appear!\n\n```ruby\nrequire 'evil_seed'\nEvilSeed.dump('path/to/new_dump.sql')\n```\n\n### Caveats, tips, and tricks\n\n 1. Specify `root`s for dictionaries and system-wide models like `Role` at the top without constraints and with all associations excluded.\n\n 2. Use `exclude` aggressively. You will be amazed, how much your app's models graph is connected. This, in conjunction with the fact that this gem traverses associations in deep-first fashion, sometimes leads to unwanted results: some records will get into dump even if you don't want them.\n\n 3. Look at the resulted dump: there are some useful debug comments.\n\n## Database compatibility\n\nThis gem has been tested against:\n\n - PostgreSQL: any version that works with ActiveRecord should work\n - MySQL: any version that works with ActiveRecord should work\n - SQLite: 3.7.11 or newer is required (with support for inserting multiple rows at a time)\n\n## Restoring dump\n\nResulting dump is a plain SQL file: you can restore it using any SQL client like `psql`, `mysql`, `sqlite3`, etc.\n\nIf you need to do it from Ruby, you can use the following code:\n\n```ruby\nActiveRecord::Base.connection.execute(File.read('path/to/new_dump.sql'))\n```\n\n### Restoration tips and tricks\n\n 1. Reset primary key sequences after restoration, so default seeds can be generated afterwards and your app will work as expected:\n\n    ```ruby\n    ActiveRecord::Base.connection.tables.each do |table|\n      ActiveRecord::Base.connection.reset_pk_sequence!(table)\n    end\n    ```\n\n 2. To restore dumps with circular dependencies between records in PostgreSQL you can make all foreign keys deferrable beforehand (by default they are not) and restore the dump in a transaction with all foreign keys deferred.\n\n    \u003cdetails\u003e\n    \u003csummary\u003eCode to defer, restore, undefer:\u003c/summary\u003e\n\n    ```ruby\n    connection = ActiveRecord::Base.connection\n\n    # Convert all foreign keys to deferrable to handle circular dependencies\n    transaction do\n      connection.tables.each do |table|\n        connection.foreign_keys(table).each do |fk|\n          connection.execute \u003c\u003c~SQL.squish\n            ALTER TABLE #{connection.quote_table_name(table)}\n            ALTER CONSTRAINT #{connection.quote_table_name(fk.options[:name])}\n            NOT DEFERRABLE\n          SQL\n        end\n      end\n    end\n\n    # Load the dump\n    connection.transaction do\n      connection.execute(\"SET CONSTRAINTS ALL DEFERRED\")\n      connection.execute(File.read(filepath))\n    end\n\n\n    # Convert all foreign keys back to not deferrable\n    # See https://begriffs.com/posts/2017-08-27-deferrable-sql-constraints.html#reasons-not-to-defer\n    connection.transaction do\n      connection.tables.each do |table|\n        connection.foreign_keys(table).each do |fk|\n          connection.execute \u003c\u003c~SQL.squish\n            ALTER TABLE #{connection.quote_table_name(table)}\n            ALTER CONSTRAINT #{connection.quote_table_name(fk.options[:name])}\n            NOT DEFERRABLE\n          SQL\n        end\n      end\n    end\n    ```\n    \u003c/details\u003e\n\n## FIXME (help wanted)\n\n 1. `has_and_belongs_to_many` associations are traversed in a bit nonintuitive way for end user:\n\n    Association path for `User.has_and_belongs_to_many :roles` is `user.users_roles.role`, but should be `user.roles`\n\n 2. Test coverage is poor\n\n 3. Some internal refactoring is required\n\n\n## Standalone usage\n\nIf you want to use it as a standalone application, you can place exerything in a single file like this:\n\n```ruby\n#!/usr/bin/env ruby\n\nrequire 'bundler/inline'\n\ngemfile do\n  source 'https://rubygems.org'\n  gem 'activerecord'\n  gem 'evil-seed'\n  gem 'mysql2'\nend\n\n# Describe your database layout with ActiveRecord models.\n# See http://guides.rubyonrails.org/active_record_basics.html\n\nclass Category \u003c ActiveRecord::Base\n  has_many :translations, class_name: \"Category::Translation\"\nend\n\nclass Category::Translation \u003c ActiveRecord::Base\n  belongs_to :category, inverse_of: :translations\nend\n\n# Configure evil-seed itself\nEvilSeed.configure do |config|\n  config.root(\"Category\", \"id \u003c ?\", 1000)\nend\n\n# Connect to your database.\n# See http://guides.rubyonrails.org/configuring.html#configuring-a-database)\nActiveRecord::Base.establish_connection(ENV.fetch(\"DATABASE_URL\"))\n\n# Create dump in dump.sql file in the same directory as this script\nEvilSeed.dump(File.join(__dir__, \"dump.sql\").to_s)\n```\n\nAnd launch it like so:\n\n```sh\nDATABASE_URL=mysql2://user:pass@host/db ruby path/to/your/script.rb\n```\n\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/palkan/evil-seed.\n\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilmartians%2Fevil-seed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilmartians%2Fevil-seed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilmartians%2Fevil-seed/lists"}