{"id":15048338,"url":"https://github.com/github/replicate","last_synced_at":"2025-10-04T10:30:35.005Z","repository":{"id":65974896,"uuid":"86614144","full_name":"github/replicate","owner":"github","description":"Dump and load relational objects between Ruby environments.","archived":true,"fork":true,"pushed_at":"2018-09-14T18:29:34.000Z","size":304,"stargazers_count":13,"open_issues_count":2,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-09-29T00:40:56.332Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/rtomayko/replicate","language":"Ruby","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"rtomayko/replicate","license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/github.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-29T18:07:59.000Z","updated_at":"2024-07-31T03:18:52.000Z","dependencies_parsed_at":"2023-02-19T18:01:06.714Z","dependency_job_id":null,"html_url":"https://github.com/github/replicate","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Freplicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Freplicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Freplicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/github%2Freplicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/github","download_url":"https://codeload.github.com/github/replicate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235237973,"owners_count":18958022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T21:11:00.700Z","updated_at":"2025-10-04T10:30:34.661Z","avatar_url":"https://github.com/github.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"Dump and load relational objects between Ruby environments.\n===========================================================\n\nThe project started at GitHub to simplify the process of getting real production\ndata into development and staging environments. We use it to replicate entire\nrepository data (including associated issue, pull request, commit comment, etc.\nrecords) from production to our development environments with a single command.\nIt's excessively useful for troubleshooting issues, support requests, and\nexception reports as well as for establishing real data for evaluating design\nconcepts.\n\nSynopsis\n--------\n\n### Installing\n\n    $ gem install replicate\n\n### Dumping objects\n\nEvaluate a Ruby expression, dumping all resulting objects to standard output:\n\n    $ replicate -r ./config/environment -d \"User.find(1)\" \u003e user.dump\n    ==\u003e dumped 4 total objects:\n    Profile        1\n    User           1\n    UserEmail      2\n\nThe `-r ./config/environment` option is used to require environment setup and\nmodel instantiation code needed by the ruby expression.\n\n### Dumping many objects with a dump script\n\nDump scripts are normal ruby source files evaluated in the context of the\ndumper. The `dump(object)` method is used to put objects into the dump stream.\n\n```ruby\n# config/replicate/dump-stuff.rb\nrequire 'config/environment'\n\n%w[rtomayko/tilt rtomayko/bcat].each do |repo_name|\n  repo = Repository.find_by_name_with_owner(repo_name)\n  dump repo\n  dump repo.commit_comments\n  dump repo.issues\nend\n```\n\nRun the dump script:\n\n    $ replicate -d config/replicate/dump-stuff.rb \u003e repos.dump\n    ==\u003e dumped 1479 total objects:\n    AR::Habtm                   101\n    CommitComment                95\n    Issue                       101\n    IssueComment                427\n    IssueEvent                  308\n    Label                         5\n    Language                     19\n    LanguageName                  1\n    Milestone                     3\n    Organization                  4\n    Profile                      82\n    PullRequest                  44\n    PullRequestReviewComment      8\n    Repository                   20\n    Team                          4\n    TeamMember                    6\n    User                         89\n    UserEmail                   162\n\n### Loading many objects:\n\n    $ replicate -r ./config/environment -l \u003c repos.dump\n    ==\u003e loaded 1479 total objects:\n    AR::Habtm                   101\n    CommitComment                95\n    Issue                       101\n    IssueComment                427\n    IssueEvent                  308\n    Label                         5\n    Language                     19\n    LanguageName                  1\n    Milestone                     3\n    Organization                  4\n    Profile                      82\n    PullRequest                  44\n    PullRequestReviewComment      8\n    Repository                   20\n    Team                          4\n    TeamMember                    6\n    User                         89\n    UserEmail                   162\n\n### Dumping and loading over ssh\n\n    $ remote_command=\"replicate -r /app/config/environment -d 'User.find(1234)'\"\n    $ ssh example.org \"$remote_command\" |replicate -r ./config/environment -l\n\nActiveRecord\n------------\n\nBasic support for dumping and loading ActiveRecord objects is included. The\ntests pass under ActiveRecord versions 2.2.3, 2.3.5, 2.3.14, 3.0.10, 3.1.0, and 3.2.0 under\nMRI 1.8.7 as well as under MRI 1.9.2.\n\nTo use customization macros in your models, require the replicate library after\nActiveRecord (in e.g., `config/initializers/libraries.rb`):\n\n```ruby\nrequire 'active_record'\nrequire 'replicate'\n```\n\nActiveRecord support works sensibly without customization so this isn't strictly\nnecessary to use the `replicate` command. The following sections document the\navailable customization macros.\n\n### Association Dumping\n\nThe baked in support adds some more or less sensible default behavior for all\nsubclasses of `ActiveRecord::Base` such that dumping an object will bring in\nobjects related via `belongs_to` and `has_one` associations.\n\nUnlike 1:1 associations, `has_many` and `has_and_belongs_to_many` associations\nare not automatically included. Doing so would quickly lead to the entire\ndatabase being sucked in. It can be useful to mark specific associations for\nautomatic inclusion using the `replicate_associations` macro. For instance,\nto always include `EmailAddress` records belonging to a `User`:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  belongs_to :profile\n  has_many   :email_addresses\n\n  replicate_associations :email_addresses\nend\n```\n\nYou may also do this by passing an option in your dump script:\n\n```ruby\ndump User.all, :associations =\u003e [:email_addresses]\n```\n\n### Natural Keys\n\nBy default, the loader attempts to create a new record with a new primary key id\nfor all objects. This can lead to unique constraint errors when a record already\nexists with matching attributes. To update existing records instead of\ncreating new ones, define a natural key for the model using the `replicate_natural_key`\nmacro:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  belongs_to :profile\n  has_many   :email_addresses\n\n  replicate_natural_key :login\n  replicate_associations :email_addresses\nend\n\nclass EmailAddress \u003c ActiveRecord::Base\n  belongs_to :user\n  replicate_natural_key :user_id, :email\nend\n```\n\nMultiple attribute names may be specified to define a compound key. Foreign key\ncolumn attributes (`user_id`) are often included in natural keys.\n\n### Omission of attributes and associations\n\nYou might want to exclude some attributes or associations from being dumped. For\nthis, use the replicate_omit_attributes macro:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  has_one    :profile\n\n  replicate_omit_attributes :created_at, :profile\nend\n```\n\nYou can omit belongs_to associations by omitting the foreign key column.\n\nYou may also do this by passing an option in your dump script:\n\n```ruby\ndump User.all, :omit =\u003e [:profile]\n```\n\n### Validations and Callbacks\n\n__IMPORTANT:__ All ActiveRecord validations and callbacks are disabled on the\nloading side. While replicate piggybacks on AR for relationship information and\nuses `ActiveRecord::Base#save` to write objects to the database, it's designed\nto act as a simple dump / load tool.\n\nIt's sometimes useful to run certain types of callbacks on replicate. For\ninstance, you might want to create files on disk or load information into a\nseparate data store any time an object enters the database. The best way to go\nabout this currently is to override the model's `load_replicant` class method:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  def self.load_replicant(type, id, attrs)\n    id, object = super\n    object.register_in_redis\n    object.some_other_callback\n    [id, object]\n  end\nend\n```\n\nThis interface will be improved in future versions.\n\nCustom Objects\n--------------\n\nOther object types may be included in the dump stream so long as they implement\nthe `dump_replicant` and `load_replicant` methods.\n\n### dump_replicant\n\nThe dump side calls `#dump_replicant(dumper, opts={})` on each object. The method must\ncall `dumper.write()` with the class name, id, and hash of primitively typed\nattributes for the object:\n\n```ruby\nclass User\n  attr_reader   :id\n  attr_accessor :name, :email\n\n  def dump_replicant(dumper, opts={})\n    attributes = { 'name' =\u003e name, 'email' =\u003e email }\n    dumper.write self.class, id, attributes, self\n  end\nend\n```\n\n### load_replicant\n\nThe load side calls `::load_replicant(type, id, attributes)` on the class to\nload each object into the current environment. The method must return an\n`[id, object]` tuple:\n\n```ruby\nclass User\n  def self.load_replicant(type, id, attributes)\n    user = User.new\n    user.name  = attributes['name']\n    user.email = attributes['email']\n    user.save!\n    [user.id, user]\n  end\nend\n```\n\nHow it works\n------------\n\nThe dump format is designed for streaming relational data. Each object is\nencoded as a `[type, id, attributes]` tuple and marshalled directly onto the\nstream. The `type` (class name string) and `id` must form a distinct key when\ncombined, `attributes` must consist of only string keys and simply typed values.\n\nRelationships between objects in the stream are managed as follows:\n\n - An object's attributes may encode references to objects that precede it\n   in the stream using a simple tuple format: `[:id, 'User', 1234]`.\n\n - The dump side ensures that objects are written to the dump stream in\n   \"reference order\" such that when an object A includes a reference attribute\n   to an object B, B is guaranteed to arrive before A.\n\n - The load side maintains a mapping of ids from the dumping system to the newly\n   replicated objects on the loading system. When the loader encounters a\n   reference value `[:id, 'User', 1234]` in an object's attributes, it converts it\n   to the load side id value.\n\nDumping and loading happens in a streaming fashion. There is no limit on the\nnumber of objects included in the stream.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Freplicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithub%2Freplicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithub%2Freplicate/lists"}