{"id":13483769,"url":"https://github.com/soundcloud/lhm","last_synced_at":"2025-05-14T02:08:57.768Z","repository":{"id":1463602,"uuid":"1700528","full_name":"soundcloud/lhm","owner":"soundcloud","description":"Online MySQL schema migrations","archived":false,"fork":false,"pushed_at":"2023-08-30T11:53:18.000Z","size":518,"stargazers_count":1843,"open_issues_count":36,"forks_count":195,"subscribers_count":198,"default_branch":"master","last_synced_at":"2025-05-04T13:18:13.956Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soundcloud.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2011-05-04T09:52:09.000Z","updated_at":"2025-04-29T19:17:13.000Z","dependencies_parsed_at":"2023-07-05T19:18:31.576Z","dependency_job_id":"48d7ddb2-c415-4717-ad66-9f7c66de6632","html_url":"https://github.com/soundcloud/lhm","commit_stats":{"total_commits":295,"total_committers":44,"mean_commits":6.704545454545454,"dds":0.7898305084745763,"last_synced_commit":"50907121eee514649fa944fb300d5d8a64fa873e"},"previous_names":["soundcloud/large-hadron-migrator"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Flhm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Flhm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Flhm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Flhm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soundcloud","download_url":"https://codeload.github.com/soundcloud/lhm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254053223,"owners_count":22006717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:15.100Z","updated_at":"2025-05-14T02:08:52.756Z","avatar_url":"https://github.com/soundcloud.png","language":"Ruby","readme":"# Large Hadron Migrator [![Build Status][5]][4]\n\nRails style database migrations are a useful way to evolve your data schema in\nan agile manner. Most Rails projects start like this, and at first, making\nchanges is fast and easy.\n\nThat is until your tables grow to millions of records. At this point, the\nlocking nature of `ALTER TABLE` may take your site down for an hour or more\nwhile critical tables are migrated. In order to avoid this, developers begin\nto design around the problem by introducing join tables or moving the data\ninto another layer. Development gets less and less agile as tables grow and\ngrow. To make the problem worse, adding or changing indices to optimize data\naccess becomes just as difficult.\n\n\u003e Side effects may include black holes and universe implosion.\n\nThere are few things that can be done at the server or engine level. It is\npossible to change default values in an `ALTER TABLE` without locking the\ntable. The InnoDB Plugin provides facilities for online index creation, which\nis great if you are using this engine, but only solves half the problem.\n\nAt SoundCloud we started having migration pains quite a while ago, and after\nlooking around for third party solutions, we decided to create our\nown. We called it Large Hadron Migrator, and it is a gem for online\nActiveRecord migrations.\n\n![LHC](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)\n\n[The Large Hadron collider at CERN](http://en.wikipedia.org/wiki/Large_Hadron_Collider)\n\n## The idea\n\nThe basic idea is to perform the migration online while the system is live,\nwithout locking the table. In contrast to [OAK][0] and the\n[facebook tool][1], we only use a copy table and triggers.\n\nThe Large Hadron is a test driven Ruby solution which can easily be dropped\ninto an ActiveRecord migration. It presumes a single auto\nincremented numerical primary key called id as per the Rails convention. Unlike\nthe [twitter solution][2], it does not require the presence of an indexed\n`updated_at` column.\n\n## Requirements\n\nLhm currently only works with MySQL databases and requires an established\nActiveRecord connection.\n\nIt is compatible and [continuously tested][4] with MRI 2.0.x, 2.1.x,\nActiveRecord 3.2.x and 4.x (mysql and mysql2 adapters).\n\n## Limitations\n\nDue to the Chunker implementation, Lhm requires that the table to migrate has\na single integer numeric key column called `id`.\n\nAnother note about the Chunker, it performs static sized row copies against the `id`\ncolumn.  Therefore sparse assignment of `id` can cause performance problems for the\nbackfills.  Typically LHM assumes that `id` is an `auto_increment` style column.\n\n## Installation\n\nInstall it via `gem install lhm` or add `gem \"lhm\"` to your Gemfile.\n\n## Usage\n\nYou can invoke Lhm directly from a plain ruby file after connecting ActiveRecord\nto your mysql instance:\n\n```ruby\nrequire 'lhm'\n\nActiveRecord::Base.establish_connection(\n  :adapter =\u003e 'mysql',\n  :host =\u003e '127.0.0.1',\n  :database =\u003e 'lhm'\n)\n\n# and migrate\nLhm.change_table :users do |m|\n  m.add_column :arbitrary, \"INT(12)\"\n  m.add_column :locale, \"VARCHAR(2) NOT NULL DEFAULT 'en'\"\n  m.add_index  [:arbitrary_id, :created_at]\n  m.ddl(\"alter table %s add column flag tinyint(1)\" % m.name)\nend\n```\n\nTo use Lhm from an ActiveRecord::Migration in a Rails project, add it to your\nGemfile, then invoke as follows:\n\n```ruby\nrequire 'lhm'\n\nclass MigrateUsers \u003c ActiveRecord::Migration\n  def self.up\n    Lhm.change_table :users do |m|\n      m.add_column :arbitrary, \"INT(12)\"\n      m.add_index  [:arbitrary_id, :created_at]\n      m.ddl(\"alter table %s add column flag tinyint(1)\" % m.name)\n    end\n  end\n\n  def self.down\n    Lhm.change_table :users do |m|\n      m.remove_index  [:arbitrary_id, :created_at]\n      m.remove_column :arbitrary\n    end\n  end\nend\n```\n\n**Note:** Lhm won't delete the old, leftover table. This is on purpose, in order\nto prevent accidental data loss.\n\n## Throttler\n\nLhm is using a throttle mechanism to read data in your original table.\n\nBy default, 40000 rows are read each 0.1 second.\n\nIf you want to change that behaviour, you can pass an instance of a throttler with the `throttler` option.\n\nIn this example, 1000 rows will be read with a 10 seconds delay between each processing:\n```ruby\nmy_throttler = Lhm::Throttler::Time.new(stride: 1000, delay: 10)\n\nLhm.change_table :users, throttler: my_throttler  do |m|\n  #\nend\n```\n\n### SlaveLag Throttler\n\nLhm uses by default the time throttler, however a better solution is to throttle the copy of the data\ndepending on the time that the slaves are behind. To use the SlaveLag throttler:\n```ruby\nLhm.change_table :users, throttler: :slave_lag_throttler  do |m|\n  #\nend\n```\n\nOr to set that as default throttler, use the following (for instance in a Rails initializer):\n```ruby\nLhm.setup_throttler(:slave_lag_throttler)\n```\n\n## Table rename strategies\n\nThere are two different table rename strategies available: LockedSwitcher and\nAtomicSwitcher.\n\nThe LockedSwitcher strategy locks the table being migrated and issues two ALTER TABLE statements.\nThe AtomicSwitcher uses a single atomic RENAME TABLE query and is the favored solution.\n\nLhm chooses AtomicSwitcher if no strategy is specified, **unless** your version of MySQL is\naffected by [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675). If your version is\naffected, Lhm will raise an error if you don't specify a strategy. You're recommended\nto use the LockedSwitcher in these cases to avoid replication issues.\n\nTo specify the strategy in your migration:\n\n```ruby\nLhm.change_table :users, :atomic_switch =\u003e true do |m|\n  # ...\nend\n```\n\n## Limiting the data that is migrated\n\nFor instances where you want to limit the data that is migrated to the new\ntable by some conditions, you may tell the migration to filter by a set of\nconditions:\n\n```ruby\nLhm.change_table(:sounds) do |m|\n  m.filter(\"inner join users on users.`id` = sounds.`user_id` and sounds.`public` = 1\")\nend\n```\n\nNote that this SQL will be inserted into the copy directly after the \"from\"\nstatement - so be sure to use inner/outer join syntax and not cross joins. These\nconditions will not affect the triggers, so any modifications to the table\nduring the run will happen on the new table as well.\n\n## Cleaning up after an interrupted Lhm run\n\nIf an Lhm migration is interrupted, it may leave behind the temporary tables\nand/or triggers used in the migration. If the migration is re-started, the\nunexpected presence of these tables will cause an error.\n\nIn this case, `Lhm.cleanup` can be used to drop any orphaned Lhm temporary tables or triggers.\n\nTo see what Lhm tables/triggers are found:\n\n```ruby\nLhm.cleanup\n```\n\nTo remove any Lhm tables/triggers found:\n```ruby\nLhm.cleanup(:run)\n```\n\nOptionally only remove tables up to a specific Time, if you want to retain previous migrations.\n\nRails:\n```ruby\nLhm.cleanup(:run, until: 1.day.ago)\n```\n\nRuby:\n```ruby\nLhm.cleanup(:run, until: Time.now - 86400)\n```\n\n## Contributing\n\nFirst, get set up for local development:\n\n    git clone git://github.com/soundcloud/lhm.git\n    cd lhm\n\nTo run the tests, follow the instructions on [spec/README](https://github.com/soundcloud/lhm/blob/master/spec/README.md).\n\nWe'll check out your contribution if you:\n\n  * Provide a comprehensive suite of tests for your fork.\n  * Have a clear and documented rationale for your changes.\n  * Package these up in a pull request.\n\nWe'll do our best to help you out with any contribution issues you may have.\n\n## License\n\nThe license is included as LICENSE in this directory.\n\n## Similar solutions\n\n  * [OAK: online alter table][0]\n  * [Facebook][1]\n  * [Twitter][2]\n  * [pt-online-schema-change][3]\n\n[0]: http://openarkkit.googlecode.com\n[1]: http://www.facebook.com/note.php?note\\_id=430801045932\n[2]: https://github.com/freels/table_migrator\n[3]: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html\n[4]: https://travis-ci.org/soundcloud/lhm\n[5]: https://travis-ci.org/soundcloud/lhm.svg?branch=master\n","funding_links":[],"categories":["Ruby","Database Tools"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Flhm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoundcloud%2Flhm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Flhm/lists"}